Although human twin studies have revealed the combined contribution of heritable and environmental factors in shaping immune system variability in blood, the contribution of these factors to immune system variability in tissues remains unexplored. The human uterus undergoes constant regeneration and is exposed to distinct environmental factors. To assess uterine immune system variation, we performed a system-level analysis of endometrial and peripheral blood immune cells in monozygotic twins. Although most immune cell phenotypes in peripheral blood showed high genetic heritability, more variation was found in endometrial immune cells, indicating a stronger influence by environmental factors. Cytomegalovirus infection was identified to influence peripheral blood immune cell variability but had limited effect on endometrial immune cells. Instead, hormonal contraception shaped the local endometrial milieu and immune cell composition with minor influence on the systemic immune system. These results highlight that the magnitude of human immune system variation and factors influencing it can be tissue specific.