As of May 2023, the public health emergency of COVID-19 was lifted across the globe. However, SARS-CoV-2 infections continue to be recorded worldwide. This situation has been attributed to the ability of the virus to evade host immune responses including neutralizing antibody-derived Immunity. The vast majority of antibody escape mutations have been associated with the S1 subunit of the spike protein, especially the Receptor Binding Domain (RBD) but also the N-terminal Domain (NTD). The other region of the spike, the S2 subunit, is the most conserved region amongst coronaviruses. We hypothesized that S2-specific antibody responses are suboptimal in vaccinated and SARS-CoV-2 infected patients resulting in an ineffective neutralization of distant coronaviruses. Here, we analyzed S2-specific antibody responses SARS-CoV-2-infected individuals, including a mixed cohort of those with and without immunosuppression and prior vaccination. We found that S2-specific antibody responses are generally lower than S1-specific antibody responses. Furthermore, we observed in immunocompetent individuals that S1 and S2-specific antibody responses are both positively correlated with Wuhan, Omicron, SARS-CoV and W1V1-CoV pseudovirus neutralization. Among the immunocompromised patients, S1-specific antibody responses were rarely correlated with pseudovirus neutralization in contrast to S2-specific antibody responses which frequently correlated with pseudovirus neutralization. These data highlight the potential of the S2-subunit as an ideal target for induction of cross-neutralizing antibody immunity against divergent coronaviruses.