Starburst galaxies are well-motivated astrophysical emitters of high-energy gamma rays. They are well-known cosmic-ray "reservoirs," thanks to their expected large magnetic field turbulence which confine high-energy protons for ∼10^{5} years. Over such long times, cosmic-ray transport can be significantly affected by scatterings with sub-GeV dark matter. Here we point out that this scattering distorts the cosmic-ray spectrum, and the distortion can be indirectly observed by measuring the gamma rays produced by cosmic rays via hadronic collisions. Present gamma-ray data show no sign of such a distortion, leading to stringent bounds on the cross section between protons and dark matter. These are highly complementary with current bounds and have large room for improvement with the future gamma-ray measurements in the 0.1-10 TeV range from the Cherenkov Telescope Array, which can strengthen the limits by as much as 2 orders of magnitude.