Image inversion interferometry can measure the separation of two incoherent point sources at or near the quantum limit. This technique has the potential to improve upon current state-of-the-art imaging technologies, with applications ranging from microbiology to astronomy. However, unavoidable aberrations and imperfections in real systems may prevent inversion interferometry from providing an advantage for real-world applications. Here, we numerically study the effects of realistic imaging system imperfections on the performance of image inversion interferometry, including common phase aberrations, interferometer misalignment, and imperfect energy splitting within the interferometer. Our results suggest that image inversion interferometry retains its superiority to direct detection imaging for a wide range of aberrations, so long as pixelated detection is used at the interferometer outputs. This study serves as a guide for the system requirements needed to achieve sensitivities beyond the limits of direct imaging, and further elucidates the robustness of image inversion interferometry to imperfections. These results are critical for the design, construction, and use of future imaging technologies performing at or near the quantum limit of source separation measurements.