We report on a kHz, mJ-level, carrier-envelope phase (CEP)-stable ultrabroadband optical parametric chirped-pulse amplifier (OPCPA) at 2.1-μm wavelength, pumped by a high-energy, 14 ps, cryogenic Yb:YAG pump laser, and its application to high-order harmonic generation (HHG) with Xe. The pre-amplifier chain is pumped by a 12-ps Nd:YLF pump laser and both pump lasers are optically synchronized to the signal pulse of the OPCPA. An amplified pulse energy of 0.85 mJ was obtained at the final OPCPA stage with good beam profile. The pulse is compressed to 4.5 optical cycles (<32 fs) with a spectral bandwidth of 474 nm supporting 3.5 optical cycles. The CEP stability was measured to be 194 mrad and the super-fluorescence noise is estimated to be ~9%. First HHG results are demonstrated with Xe showing significant cutoff extension to >85 eV with an efficiency of ~10-10 per harmonic, limited by the maximum gas pressure and flow into the chamber. This demonstrates the potential of this 2.1-μm source for scaling of photon energy and flux in the water-window range when applied to Ne and He at kHz repetition rate.