The impact of large-scale chromosomal rearrangements, such as fusions and fissions, on speciation is a long-standing conundrum. We assessed whether bursts of change in chromosome numbers resulting from chromosomal fusion or fission are related to increased speciation rates in Erebia, one of the most species-rich and karyotypically variable butterfly groups. We established a genome-based phylogeny and used state-dependent birth-death models to infer trajectories of karyotype evolution. We demonstrated that rates of anagenetic chromosomal changes (i.e., along phylogenetic branches) exceed cladogenetic changes (i.e., at speciation events), but, when cladogenetic changes occur, they are mostly associated with chromosomal fissions rather than fusions. We found that the relative importance of fusion and fission differs among Erebia clades of different ages and that especially in younger, more karyotypically diverse clades, speciation is more frequently associated with cladogenetic chromosomal changes. Overall, our results imply that chromosomal fusions and fissions have contrasting macroevolutionary roles and that large-scale chromosomal rearrangements are associated with bursts of species diversification.