Terahertz (THz) radiation promises breakthrough advances in compact advanced accelerators due to the gigavolts-per-meter fields achievable, but the challenge of maintaining overlap and synchronism between beams and short laser-generated THz pulses has so far limited interactions to the few-millimeter scale. We implement a novel scheme for simultaneous group and phase velocity matching of nearly single-cycle THz radiation with a relativistic electron beam for meter-scale inverse free-electron laser interaction in a magnetic undulator, resulting in energy modulations of up to 150 keV using modest THz pulse energies (≤1 μJ). Using this scheme, we demonstrate for the first time the use of a laser-based THz source for bunch-length compression and time-stamping of a relativistic electron beam.