Our analysis examined the impact of maternal dietary patterns and lifestyle factors on markers of fetal growth, specifically birthweight and size for gestational age (small- (SGA) or large-for-gestational age (LGA)). The Infant Feeding Practices Study II, a prospective cohort study, surveyed pregnant women during their 3rd trimester, of which a subgroup (n = 893) completed a food frequency questionnaire. Maternal dietary patterns were evaluated by diet scores (Alternative Healthy Eating Index for Pregnancy and alternate Mediterranean diet) and by carbohydrate quality (glycemic index and glycemic load). Poisson regression with robust standard errors was used to examine the relative risk of SGA and separately LGA, with dietary patterns and other lifestyle factors. Linear regression was used to determine the association of birthweight and early infant growth with better dietary patterns. Relative risk of SGA and LGA was not associated with dietary patterns. Birthweight and infant growth were not associated with maternal diet. Smoking, however, increased the risk of delivering an SGA infant (RR = 2.92, 95% CI: 1.58-5.39), while higher prepregnancy BMI increased the risk of delivering an LGA infant (RR = 1.06, 95% CI: 1.03-1.09). Future studies are needed to evaluate whether deficiencies in more specific maternal dietary nutrients play a role in fetal growth.