Chromosome segregation errors cause aneuploidy and genomic heterogeneity, which are hallmarks of cancer in humans. A persistent high frequency of these errors (chromosomal instability (CIN)) is predicted to profoundly impact tumor evolution and therapy response. It is unknown, however, how prevalent CIN is in human tumors. Using three-dimensional live-cell imaging of patient-derived tumor organoids (tumor PDOs), we show that CIN is widespread in colorectal carcinomas regardless of background genetic alterations, including microsatellite instability. Cell-fate tracking showed that, although mitotic errors are frequently followed by cell death, some tumor PDOs are largely insensitive to mitotic errors. Single-cell karyotype sequencing confirmed heterogeneity of copy number alterations in tumor PDOs and showed that monoclonal lines evolved novel karyotypes over time in vitro. We conclude that ongoing CIN is common in colorectal cancer organoids, and propose that CIN levels and the tolerance for mitotic errors shape aneuploidy landscapes and karyotype heterogeneity.