We experimentally study particle scale dynamics during segregation of a bidisperse mixture under oscillatory shear. Large and small particles show an underlying asymmetry that is dependent on the local particle concentration, with small particles segregating faster in regions of many large particles and large particles segregating slower in regions of many small particles. We quantify the asymmetry on bulk and particle scales, and capture it theoretically. This gives new physical insight into segregation and reveals a similarity with sedimentation, traffic flow, and particle diffusion.