Verbal fluency impairments are frequent in Parkinson's disease (PD) and they may be present already at early stages. Semantic fluency impairment is associated with Parkinson's disease dementia and temporal, frontal and cerebellar cortical changes. Few studies have addressed cerebral structural correlates of different verbal fluency tasks in early stage PD. We therefore studied gray matter volumes of T1-weighted MRI images using voxel-based morphometry in relation to semantic, phonemic, and alternating verbal fluency in younger (mean age <65 years), early stage (mean disease duration <3 years), non-demented PD patients (n = 28) and healthy controls (n = 27). We found a significant association between worse phonemic fluency and smaller striatal, namely right caudate gray matter volume in the PD group only (family-wise error corrected p = 0.007). Reduced semantic fluency was associated with smaller gray matter volumes in left parietal cortex (p = 0.037) and at trend level with smaller bilateral cerebellum gray matter volume across groups (p = 0.062), but not in the separate PD or control groups. There were no significant relationships between alternating fluency and gray matter volumes in the whole sample or in the groups separately. The fact that phonemic fluency, but not semantic or alternating fluency, was associated with caudate gray matter volume at early stage PD suggests that different fluency tasks rely on different neural substrates, and that language networks supporting semantic search and verbal-semantic switching are unrelated to brain gray matter volume at early disease stages in PD.