Chronic excessive alcohol intake results in alcohol-related brain damage. Many previous reports have documented alcohol-related global or local brain shrinkage or diffusional abnormalities among alcoholics and heavy to moderate drinkers; however, the influence of relatively low levels of alcohol consumption on brain structural or diffusional abnormality is unclear. We investigated structural or diffusional abnormalities related to lifetime alcohol consumption (LAC) using voxel-based morphometry (VBM) among Japanese non-alcohol-dependent individuals (114 males, 97 females). High-resolution three-dimensional magnetic resonance images and diffusion tensor imaging were acquired in all subjects. The collected images were normalized, segmented, and smoothed using SPM 5. Gray matter volume (GMV) and white matter volume (WMV) were normalized for each total intracranial volume (TIV), and partial correlation coefficients were estimated between normalized GMV or WMV and lifetime alcohol consumption (LAC) adjusted for age. To investigate regional GMV or WMV abnormalities related to LAC, multiple regression analyses were performed among regional GMV or WMV and LAC, age, and TIV. To investigate subtle regional abnormalities, multiple regression analyses were performed among fractional anisotropy (FA) or mean diffusivity (MD), and LAC and age. No LAC-related global or regional GMV or WMV abnormality or LAC-related regional FA abnormality was found among male or female subjects. Significant LAC-related MD increase was found in the right amygdala among female subjects only. The current results suggest female brain vulnerability to alcohol, and a relation between subtle abnormality in the right amygdala and alcohol misuse.