Assessing fetal development is essential to the provision of healthcare for both mothers and fetuses. In low- and middle-income countries, conditions that increase the risk of fetal growth restriction (FGR) are often more prevalent. In these regions, barriers to accessing healthcare and social services exacerbate fetal maternal health problems. One of these barriers is the lack of affordable diagnostic technologies. To address this issue, this work introduces an end-to-end algorithm applied to a low-cost, hand-held Doppler ultrasound device for estimating gestational age (GA), and by inference, FGR. The Doppler ultrasound signals used in this study were collected from 226 pregnancies (45 low birth weight at delivery) between 5 and 9 months GA by lay midwives in highland Guatemala. We designed a hierarchical deep sequence learning model with an attention mechanism to learn the normative dynamics of fetal cardiac activity in different stages of development. This resulted in a state-of-the-art GA estimation performance, with an average error of 0.79 months. This is close to the theoretical minimum for the given quantization level of one month. The model was then tested on Doppler recordings of the fetuses with low birth weight and the estimated GA was shown to be lower than the GA calculated from last menstruation. Thus, this could be interpreted as a potential sign of developmental retardation (or FGR) associated with low birth weight, and referral and intervention may be necessary.