Understanding the drivers of variability in the composition of fish assemblages across the Indo-Pacific region is crucial to support coral reef ecosystem resilience. Whilst numerous relationships and feedback mechanisms between the functional roles of coral reef fishes and reef benthic composition have been investigated, certain key groups, such as the herbivores, are widely suggested to maintain reefs in a coral-dominated state. Examining links between fishes and reef benthos is complicated by the interactions between natural processes, disturbance events and anthropogenic impacts, particularly fishing pressure. This study examined fish assemblages and associated benthic variables across five atolls within the Chagos Archipelago, where fishing pressure is largely absent, to better understand these relationships. We found high variability in fish assemblages among atolls and sites across the archipelago, especially for key groups such as a suite of grazer-detritivore surgeonfish, and the parrotfishes which varied in density over 40-fold between sites. Differences in fish assemblages were significantly associated with variable levels of both live and recently dead coral cover and rugosity. We suggest these results reflect differing coral recovery trajectories following coral bleaching events and a strong influence of 'bottom-up' control mechanisms on fish assemblages. Species level analyses revealed that Scarus niger, Acanthurus nigrofuscus and Chlorurus strongylocephalos were key species driving differences in fish assemblage structure. Clarifying the trophic roles of herbivorous and detritivorous reef fishes will require species-level studies, which also examine feeding behaviour, to fully understand their contribution in maintaining reef resilience to climate change and fishing impacts.