Cancer patients are a vulnerable population postulated to be at higher risk for severe COVID-19 infection. Increased COVID-19 morbidity and mortality in cancer patients may be attributable to age, comorbidities, smoking, healthcare exposure, and cancer treatments, and partially to the cancer itself. Most studies to date have focused on hospitalized patients with severe COVID-19, thereby limiting the generalizability and interpretability of the association between cancer and COVID-19 severity. We compared outcomes of SARS-CoV-2 infection in 323 patients enrolled prior to the pandemic in a large academic biobank (n=67 cancer patients and n=256 non-cancer patients). After adjusting for demographics, smoking status, and comorbidities, a diagnosis of cancer was independently associated with higher odds of hospitalization (OR 2.16, 95% CI 1.12-4.18) and 30-day mortality (OR 5.67, CI 1.49-21.59). These associations were primarily driven by patients with active cancer. These results emphasize the critical importance of preventing SARS-CoV-2 exposure and mitigating infection in cancer patients.