The ongoing retreat of glaciers at southern sub-polar latitudes is particularly rapid and widespread. Akin to northern sub-polar latitudes, this retreat is generally assumed to be linked to warming. However, no long-term and well-constrained glacier modeling has ever been performed to confirm this hypothesis. Here, we model the Cook Ice Cap mass balance on the Kerguelen Islands (Southern Indian Ocean, 49°S) since the 1850s. We show that glacier wastage during the 2000s in the Kerguelen was among the most dramatic on Earth. We attribute 77% of the increasingly negative mass balance since the 1960s to atmospheric drying associated with a poleward shift of the mid-latitude storm track. Because precipitation modeling is very challenging for the current generation of climate models over the study area, models incorrectly simulate the climate drivers behind the recent glacier wastage in the Kerguelen. This suggests that future glacier wastage projections should be considered cautiously where changes in atmospheric circulation are expected.