The angle-dependent cusp anomalous dimension governs divergences coming from soft gluon exchanges between heavy particles, such as top quarks. We focus on the matter-dependent contributions and compute the first truly nonplanar terms. They appear at four loops and are proportional to a quartic Casimir operator in color space. Specializing our general gauge theory result to U(1), we obtain the full QED four-loop angle-dependent cusp anomalous dimension. While more complicated functions appear at intermediate steps, the analytic answer depends only on multiple polylogarithms with singularities at fourth roots of unity. It can be written in terms of four rational structures and contains functions of up to maximal transcendental weight seven. Despite this complexity, we find that numerically the answer is tantalizingly close to the appropriately rescaled one-loop formula, over most of the kinematic range. We take several limits of our analytic result, which serves as a check and allows us to obtain new, power-suppressed terms. In the antiparallel lines limit, which corresponds to production of two massive particles at threshold, we find that the subleading power correction vanishes. Finally, we compute the quartic Casimir contribution for scalars in the loop. Taking into account a supersymmetric decomposition, we derive the first nonplanar corrections to the quark antiquark potential in maximally supersymmetric gauge theory.