Vitamin D deficiency is a common deficiency worldwide, particularly among women of reproductive age. During pregnancy, it increases the risk of immune-related diseases in offspring later in life. However, exactly how the body remembers exposure to an adverse environment during development is poorly understood. Herein, we explore the effects of prenatal vitamin D deficiency on immune cell proportions in offspring using vitamin D deficient mice established by dietary manipulation. We found that prenatal vitamin D deficiency alters immune cell proportions in offspring by changing the transcriptional properties of genes downstream of vitamin D receptor signaling in hematopoietic stem and progenitor cells of both the fetus and adults. Our results suggest the role of cellular differentiation properties of the hematopoiesis as the long-term memories of prenatal exposure at the adult stage. Moreover, further investigations of the associations between maternal vitamin D levels and cord blood immune cell profiles from 75 healthy pregnant women and their term babies also confirm that maternal vitamin D levels in the second trimester significantly affect immune cell proportions in the babies. This highlights the importance of providing vitamin D supplementation at specific stages of pregnancy.