Length and weight in infancy are associated with neurodevelopment, but less is known about growth in other anthropometric measures. In this study we analyzed how the development in length, weight, head circumference, and chest circumference over infancy is associated with motor development in early childhood, using a twin study design. Information on physical development over infancy and the age at achievement of eight developmental milestones over early childhood was collected for 370 Japanese twin pairs. Linear mixed models were used to analyze how physical development is associated with motor development between individual twins, as well as within twin pairs, adjusting the results for shared maternal and postnatal environmental factors. Delayed motor development was associated with smaller body size over infancy, and we also found some suggestive evidence that it was associated with catch-up growth as well. When studying the associations within twin pairs discordant for motor development, similar associations were found. However, chest circumference showed the most robust association within discordant twin pairs. Smaller body size and rapid catch-up growth are associated with delayed motor development. When studying these associations within twin pairs and thus adjusting the results for gestational age as well as many other maternal and postnatal environmental factors, chest circumference showed the most robust association. Chest circumference, rarely used in developed countries, can offer additional information on prenatal conditions relevant for further motor development not achieved by more traditional anthropometric measures.