Knowledge of atomic diffusion is a fundamental issue in synthesis and stability of materials. Direct studies of the elementary diffusion event, that is, how the individual atoms 'jump', are scarce, as the available techniques are limited to selected systems. Here we show how by monitoring the spatial and temporal variations of the scattered coherent X-ray intensity the diffusion of single atoms can be studied. This is demonstrated for the intermetallic alloy Cu(90)Au(10). By measuring along several directions in reciprocal space, we can elucidate the dynamical behaviour of single atoms as a function of their neighbourhood. This method, usually referred to as X-ray photon correlation spectroscopy (XPCS), does not rely on specific atomic species or isotopes and can thus be applied to almost any system. Thus, given the advent of the next-generation X-ray sources, XPCS has the potential to become the main method for quantitatively understanding diffusion on the atomic scale.