Single photon nonlinearities based on a semiconductor quantum dot in an optical microcavity are a promising candidate for integrated optical quantum information processing nodes. In practice, however, the finite quantum dot lifetime and cavity-quantum dot coupling lead to reduced fidelity. Here we show that, with a nearly polarization degenerate microcavity in the weak coupling regime, polarization pre- and postselection can be used to restore high fidelity. The two orthogonally polarized transmission amplitudes interfere at the output polarizer; for special polarization angles, which depend only on the device cooperativity, this enables cancellation of light that did not interact with the quantum dot. With this, we can transform incident coherent light into a stream of strongly correlated photons with a second-order correlation value up to 40, larger than previous experimental results, even in the strong-coupling regime. This purification technique might also be useful to improve the fidelity of quantum dot based logic gates.