We present the first measurements of the ratios of cross sections σ(pp → Z+c jet)/σ(pp → Z+jet) and σ(pp → Z+c jet)/σ(pp → Z+b jet) for the associated production of a Z boson with at least one charm or bottom quark jet. Jets have transverse momentum pT(jet)>20 GeV and pseudorapidity |η(jet)|<2.5. These cross section ratios are measured differentially as a function of jet and Z boson transverse momenta, based on 9.7 fb(-1) of pp collisions collected with the D0 detector at the Fermilab Tevatron Collider at √s=1.96 TeV. The measurements show significant deviations from perturbative QCD calculations and predictions from various event generators.