The common features of autism spectrum disorder, a highly heritable representative pervasive developmental disorder with significant heterogeneity and multiple-genetic factors, are severe dysfunction in social reciprocity, abnormalities in social brain regions, and disproportionately low probability in the female gender. Concomitantly, certain domains of mental function, such as emotional memory and social reciprocity, show a significant sex difference. In addition, recent neuroimaging studies have shown significant sexual dimorphisms in neuroanatomical correlates of social cognition. Recently, some sexually dimorphic factors, including oxytocin, vasopressin, and genes linked with the x-chromosome, have received attention because of their possible contribution to mental development especially in the social cognitive domain. Taking this evidence together, it is hypothesized that a sexually dimorphic factor associated with social reciprocity could affect characteristics of autism spectrum disorder including dysfunction in social reciprocity, abnormalities in social brain regions, and disproportionately low probability in female gender. This review article overviews sexual dimorphisms in clinical features of autism spectrum disorder, in normal social cognition, and in social brain function and structure. The association of oxytocin with sexual dimorphisms, social reciprocity, neural correlates of social cognition, and the pathogenesis of autism spectrum disorder were further summarized. Recent studies have suggested that oxytocin plays a role in social attachment in experimental animals, in enhancing social interactive ability in human adults, and in the pathogenesis of autism spectrum disorder. Thus, the ongoing accumulated evidence suggests that oxytocin deserves to be examined as a candidate that causes the sexually dimorphic aspect of human social reciprocity, social brain development and the pathogenesis of autism spectrum disorder.