Doppler echocardiography is widely used for functional assessment of heart valves such as mitral valve. In current clinical work flow, to extract Doppler measurements, the envelopes of acquired Doppler spectra are manually traced. We propose a robust algorithm for automatically tracing the envelopes of mitral valve inflow Doppler spectra, which exhibit a large amount of variations in envelope shape and image appearance due to various disease conditions, patient/sonographer/instrument differences, etc. The algorithm is learning-based and capable of fully automatic detection and segmentation of the mitral inflow structures. Experiments show that the algorithm, running within one second, yields comparable performance to experts.