Time reversal symmetry stands as a fundamental restriction on the vast majority of optical systems and devices. The reciprocal nature of Maxwell's equations in linear, time-invariant media adds complexity and scale to photonic diodes, isolators, circulators and also sets fundamental efficiency limits on optical energy conversion. Though many theoretical proposals and low frequency demonstrations of nonreciprocity exist, Faraday rotation remains the only known nonreciprocal mechanism that persists down to the atomic scale. Here, we present photon-spin-polarized stimulated Raman scattering as a new nonreciprocal optical phenomenon which has, in principle, no lower size limit. Exploiting this process, we numerically demonstrate nanoscale nonreciprocal transmission of free-space beams at near-infrared frequencies with a 250 nm thick silicon metasurface as well as a fully-subwavelength plasmonic gap nanoantenna. In revealing all-optical spin-splitting, our results provide a foundation for compact nonreciprocal communication and computing technologies, from nanoscale optical isolators and full-duplex nanoantennas to topologically-protected networks.