Quantum mechanics sets fundamental limits on how fast quantum states can be transformed in time. Two well-known quantum speed limits are the Mandelstam-Tamm and the Margolus-Levitin bounds, which relate the maximum speed of evolution to the system’s energy uncertainty and mean energy, respectively. Here, we test concurrently both limits in a multilevel system by following the motion of a single atom in an optical trap using fast matter wave interferometry. We find two different regimes: one where the Mandelstam-Tamm limit constrains the evolution at all times, and a second where a crossover to the Margolus-Levitin limit occurs at longer times. We take a geometric approach to quantify the deviation from the speed limit, measuring how much the quantum evolution deviates from the geodesic path in the Hilbert space of the multilevel system. Our results are important to understand the ultimate performance of quantum computing devices and related advanced quantum technologies.