The cerebellum contains most of the neurons in the human brain and exhibits distinctive modes of development and aging. In this work, by developing our single-cell three-dimensional (3D) genome assay-diploid chromosome conformation capture, or Dip-C-into population-scale (Pop-C) and virus-enriched (vDip-C) modes, we resolved the first 3D genome structures of single cerebellar cells, created life-spanning 3D genome atlases for both humans and mice, and jointly measured transcriptome and chromatin accessibility during development. We found that although the transcriptome and chromatin accessibility of cerebellar granule neurons mature in early postnatal life, 3D genome architecture gradually remodels throughout life, establishing ultra-long-range intrachromosomal contacts and specific interchromosomal contacts that are rarely seen in neurons. These results reveal unexpected evolutionarily conserved molecular processes that underlie distinctive features of neural development and aging across the mammalian life span.