Background: Considering the large number of patients with pulmonary symptoms admitted to the emergency department daily, it is essential to diagnose them correctly. It is necessary to quickly solve the differential diagnosis between COVID-19 and typical bacterial pneumonia to address them with the best management possible. In this setting, an artificial intelligence (AI) system can help radiologists detect pneumonia more quickly.
Methods: We aimed to test the diagnostic performance of an AI system in detecting COVID-19 pneumonia and typical bacterial pneumonia in patients who underwent a chest X-ray (CXR) and were admitted to the emergency department. The final dataset was composed of three sub-datasets: the first included all patients positive for COVID-19 pneumonia (n = 1140, namely "COVID-19+"), the second one included all patients with typical bacterial pneumonia (n = 500, "pneumonia+"), and the third one was composed of healthy subjects (n = 1000). Two radiologists were blinded to demographic, clinical, and laboratory data. The developed AI system was used to evaluate all CXRs randomly and was asked to classify them into three classes. Cohen's κ was used for interrater reliability analysis. The AI system's diagnostic accuracy was evaluated using a confusion matrix, and 95%CIs were reported as appropriate.
Results: The interrater reliability analysis between the most experienced radiologist and the AI system reported an almost perfect agreement for COVID-19+ (κ = 0.822) and pneumonia+ (κ = 0.913). We found 96% sensitivity (95% CIs = 94.9-96.9) and 79.8% specificity (76.4-82.9) for the radiologist and 94.7% sensitivity (93.4-95.8) and 80.2% specificity (76.9-83.2) for the AI system in the detection of COVID-19+. Moreover, we found 97.9% sensitivity (98-99.3) and 88% specificity (83.5-91.7) for the radiologist and 97.5% sensitivity (96.5-98.3) and 83.9% specificity (79-87.9) for the AI system in the detection of pneumonia+ patients. Finally, the AI system reached an accuracy of 93.8%, with a misclassification rate of 6.2% and weighted-F1 of 93.8% in detecting COVID+, pneumonia+, and healthy subjects.
Conclusions: The AI system demonstrated excellent diagnostic performance in identifying COVID-19 and typical bacterial pneumonia in CXRs acquired in the emergency setting.
Keywords: COVID-19; SARS-CoV-2; artificial intelligence; chest X-ray.