Fungal centromeres are clustered near microtubule organizing centers to help adopt the Rabl chromosomal organization. The role of centromere clustering in driving large-scale changes in structural and functional chromatin assembly remains unclear. Here, using Hi-C and superresolution microscopy, we show that cell cycle-dependent centromere declustering and clustering states in drive global changes in the 3D genome architecture. Centromeres and telomeres are scattered around the nuclear periphery at interphase, and this arrangement constrains the interarm interactions within a chromosome, providing a unique interphase chromosome organization. Moreover, centromeres and telomeres are organized as separate compartments, segregating them from active euchromatic regions. Polymer modeling reveals that the transition from the unclustered to clustered centromere state during the cell cycle involves changes from a globular to an elongated chromosome architecture. Strikingly, while clustered centromeric regions replicate early in most yeasts, centromeres replicate late in S-phase, hinting at a possible link between centromere clustering dynamics and DNA replication timing. Overall, our study uncovers several unique organizational principles governing the dynamic genome architecture in an evolutionarily diverged basidiomycete yeast.