Longitudinal analysis of magnetic resonance images of the human brain provides knowledge of brain changes during both normal aging as well as the progression of many diseases. Previous longitudinal segmentation methods have either ignored temporal information or have incorporated temporal consistency constraints within the algorithm. In this work, we assume that some anatomical brain changes can be explained by temporal transitions in image intensities. Once the images are aligned in the same space, the intensities of each scan at the same voxel constitute a temporal (or 4D) intensity trend at that voxel. Temporal intensity variations due to noise or other artifacts are corrected by a 4D intensity-based filter that smooths the intensity values where appropriate, while preserving real anatomical changes such as atrophy. Here smoothing refers to removal of sudden changes or discontinuities in intensities. Images processed with the 4D filter can be used as a pre-processing step to any segmentation method. We show that such a longitudinal pre-processing step produces robust and consistent longitudinal segmentation results, even when applying 3D segmentation algorithms. We compare with state-of-the-art 4D segmentation algorithms. Specifically, we experimented on three longitudinal datasets containing 4-12 time-points, and showed that the 4D temporal filter is more robust and has more power in distinguishing between healthy subjects and those with dementia, mild cognitive impairment, as well as different phenotypes of multiple sclerosis.