Using Predictive Models to Improve Care for Patients Hospitalized with COVID-19 [Internet]

Excerpt

Background: The COVID-19 pandemic has presented significant burdens for the United States. The purpose of this project was to generate evidence for critical questions of clinical decision-making by identifying clinical and social risk factors for COVID-19 outcomes as well as developing and validating electronic health record (EHR) data-based prediction models for patient outcomes.

Objectives: This project was conducted in close collaboration with the INSIGHT clinical research network (CRN), which is part of PCORnet®, the National Patient Centered Clinical Research Network. This project had 3 aims:

1. Aim 1: Predict the intensive care unit (ICU) need among patients hospitalized for COVID-19.

2. Aim 2: Predict the risk of mortality among patients hospitalized for COVID-19.

3. Aim 3: Predict the course and outcome of intubation among patients hospitalized for COVID-19.

Given the rapid shift in ICU designations in New York City (NYC) hospitals during the pandemic, the ICU label in the EHR data may not accurately and comprehensively identify all critical care patients. We therefore combined aim 1 and 3 and used intubation as a proxy for ICU need.

Methods: Our revised 2 specific aims of this project are (1) to predict the risk of intubation among patients hospitalized for COVID-19 and (2) to predict the risk of mortality among patients hospitalized for COVID-19. In this project, we used a COVID-19 research database developed by the INSIGHT CRN. The database includes EHR data from 5 health systems: Weill Cornell Medicine, Columbia University, Montefiore, Mount Sinai Hospital, and New York University. For aim A, we developed logistic regression models, random forests, and classification and regression tree (CART) models to predict the need for intubation from COVID-19 using data from the beginning of the pandemic (March 1, 2020, through February 8, 2021). We considered a broader set of variables as candidate predictors, including patient demographics, baseline comorbidities, and presenting laboratory tests. Our main predictors were demographic characteristics (eg, age, sex, race, ethnicity), clinical comorbidities (eg, hypertension, hyperlipidemia, chronic obstructive pulmonary disease [COPD], cancer, coronary artery disease, heart failure, asthma, diabetes), and vital signs (eg, body mass index, systolic and diastolic blood pressure). We also measured the time course of the pandemic as the number of weeks from March 2020 (the beginning of the pandemic in NYC) and measured social vulnerability as the quintiles of the Social Deprivation Index (SDI) at the zip code level among NYC residents. For aim B, we constructed logistic regression models, random forests, and CART models to predict morality. The analysis used the same demographic characteristics (eg, age, sex, race, ethnicity), clinical comorbidities (eg, hypertension, hyperlipidemia, chronic obstructive coronary disease, cancer, coronary artery disease, heart failure, asthma, diabetes), and vital signs (eg, body mass index, systolic and diastolic blood pressure) as in aim A as key predictors. Across aims A and B, we derived and validated 4 biologically distinct subphenotypes of patients with COVID-19 by using clustering analysis to understand the heterogeneity of clinical characteristics among COVID-19. We also examined the variation in our key clinical outcomes (eg, in-hospital mortality and intubation) across subphenotypes.

Results:

1. Aim A: A total of 30 016 patients from the INSIGHT COVID-19 database were used in the analysis, including 11 254 patients who were seen in the emergency department and discharged home and 18 762 hospitalized patients (2902 patients were intubated and 3554 patients died during the study period). The prediction models using logistic regressions and random forests for intubation performed. The range of the area under the receiver operating characteristic curve (AUROC) across the folds was 0.66 to 0.74 for logistic regression, 0.66 to 0.73 for random forests, and 0.53 to 0.54 for CART. Our logistic models were well calibrated, as indicated by the low Brier scores (<0.25), and the joint test of the hypotheses that showed the intercept = 0 and slope = 1 for the calibration curves for all models (P > .99 for all models). Including time (ie, number of weeks from March 1, 2020, to the date of the COVID-19 encounter) and interactions of time, with main predictors modestly improving the prediction accuracy of intubation using logistic regression and random forests. Including SDI quintiles did not improve the prediction accuracy of intubation for any method. We identified 4 clinically distinct subphenotypes, and our analysis demonstrated variations in the rates of intubation across these subphenotypes. Subphenotype I consisted of more young and female patients and had the lowest rate of intubation. Subphenotype IV included patients who were older and predominantly male, with more abnormal values across all clinical variables; this group had the highest intubation rate.

2. Aim B: The models using logistic regressions and random forests performed well in predicting in-hospital mortality. The AUROC across the folds ranged from 0.78 to 0.85 for logistic regression, 0.79 to 0.85 for random forests, and 0.64 to 0.68 for CART. Including time and interactions of time with the main predictors slightly improved the prediction accuracy using logistic regression and random forests. Including SDI quintiles did not improve prediction accuracy. Similarly, mortality varied across subphenotypes. Subphenotype I had the lowest rate of mortality, while subphenotype IV had the highest rate of mortality.

Conclusions: Our models using logistic regressions and random forests to predict in-hospital mortality and intubation showed good performance in a large sample of patients with COVID-19 in NYC. We validated the prediction models across different courses of the pandemic and across different patient neighborhood socioeconomic status. We also derived 4 subphenotypes to understand the clinical heterogeneity of patients with COVID-19 and examined demographics, clinical characteristics, and COVID-19 outcomes across these subphenotypes. These findings provide important evidence to improve outcomes among patients with COVID-19. The prediction models and subphenotypes can be implemented in health systems using their real-time EHR data and identify patients at high risk for adverse outcomes. Clinicians can target early interventions to these patients and perhaps improve patient outcomes.

Limitations: Although we drew on a robust COVID-19 patient cohort from 5 major health systems in NYC, findings may not be generalizable to other patients in the NYC area or patients in other parts of the country because of the evolution of the virus. We were not able to extract presenting symptoms and other nonroutinely collected clinical data when we developed prediction models, and we did not include clinical encounters at health systems outside the INSIGHT CRN in our clinical analyses. We used data from patients with COVID-19 in the first 2 waves of the pandemic in NYC. Therefore, findings may not be generalizable to the later waves. Our prediction models need to be validated using patients with COVID-19 from later waves. Furthermore, our study used data when vaccines were not widely available, and our models should be reevaluated and updated in a vaccinated cohort.