In atomic magnetometers, the vector AC-Stark shift associated with circularly polarized light generates spatially varying effective magnetic fields, which limit the magnetometer response and serve as sources of noise. We describe a scheme whereby optically pumping a small subvolume of the magnetometer cell and relying on diffusion to transport polarized atoms allows a magnetometer to be operated with minimal sensitivity to the AC-Stark field.