Mild cognitive impairment (MCI) and Alzheimer's disease (AD) are associated with a progressive loss of cognitive abilities. In the present report, we assessed the relationship of memory and executive function with brain structure in a sample of 810 Alzheimer's Disease Neuroimaging Initiative (ADNI) participants, including 188 AD, 396 MCI, and 226 healthy older adults (HC). Composite scores of memory (ADNI-Mem) and executive function (ADNI-Exec) were generated by applying modern psychometric theory to item-level data from ADNI's neuropsychological battery. We performed voxel-based morphometry (VBM) and surface-based association (SurfStat) analyses to evaluate relationships of ADNI-Mem and ADNI-Exec with grey matter (GM) density and cortical thickness across the whole brain in the combined sample and within diagnostic groups. We observed strong associations between ADNI-Mem and medial and lateral temporal lobe atrophy. Lower ADNI-Exec scores were associated with advanced GM and cortical atrophy across broadly distributed regions, most impressively in the bilateral parietal and temporal lobes. We also evaluated ADNI-Exec adjusted for ADNI-Mem, and found associations with GM density and cortical thickness primarily in the bilateral parietal, temporal, and frontal lobes. Within-group analyses suggest these associations are strongest in patients with MCI and AD. The present study provides insight into the spatially unbiased associations between brain atrophy and memory and executive function, and underscores the importance of structural brain changes in early cognitive decline.