We report on the optical spectroscopy of a single InAs/GaAs quantum dot doped with a single Mn atom in a longitudinal magnetic field of a few Tesla. Our findings show that the Mn impurity is a neutral acceptor state A0 whose effective spin J=1 is significantly perturbed by the quantum dot potential and its associated strain field. The spin interaction with photocarriers injected in the quantum dot is shown to be ferromagnetic for holes, with an effective coupling constant of a few hundreds of mueV, but vanishingly small for electrons.