Design and analysis choices for safety surveillance evaluations need to be tuned to the specifics of the hypothesized drug-outcome association

Susan Gruber; Aloka Chakravarty; Susan R Heckbert; Mark Levenson; David Martin; Jennifer C Nelson; Bruce M Psaty; Simone Pinheiro; Christian G Reich; Sengwee Toh; Alexander M Walker

doi:10.1002/pds.4065

Design and analysis choices for safety surveillance evaluations need to be tuned to the specifics of the hypothesized drug-outcome association

Pharmacoepidemiol Drug Saf. 2016 Sep;25(9):973-81. doi: 10.1002/pds.4065. Epub 2016 Jul 14.

Authors

Susan Gruber^{1

2}, Aloka Chakravarty³, Susan R Heckbert^{4

5}, Mark Levenson³, David Martin⁶, Jennifer C Nelson⁷, Bruce M Psaty^{5

8}, Simone Pinheiro⁹, Christian G Reich¹⁰, Sengwee Toh², Alexander M Walker¹¹

Affiliations

¹ Reagan-Udall Foundation for the FDA, Innovation in Medical Evidence Development and Surveillance, Washington, DC, USA.
² Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA, USA.
³ Office of Biostatistics, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA.
⁴ Cardiovascular Health Research Unit, Department of Epidemiology, University of Washington, Seattle, WA, USA.
⁵ Group Health Research Institute, Group Health Cooperative, Seattle, WA, USA.
⁶ Office of the Center Director, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA.
⁷ Biostatistics Unit, Group Health Research Institute, Department of Biostatistics, University of Washington, Seattle, WA, USA.
⁸ Cardiovascular Health Research Unit, Departments of Medicine, Epidemiology, and Health Services, University of Washington, Seattle, WA, USA.
⁹ Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA.
¹⁰ RWE Systems, IMS Health, Burlington, MA, USA.
¹¹ WHISCON, Newton, MA, USA.

PMID: 27418432
DOI: 10.1002/pds.4065

Abstract

Background: We reviewed the results of the Observational Medical Outcomes Research Partnership (OMOP) 2010 Experiment in hopes of finding examples where apparently well-designed drug studies repeatedly produce anomalous findings. OMOP had applied thousands of designs and design parameters to 53 drug-outcome pairs across 10 electronic data resources. Our intent was to use this repository to elucidate some sources of error in observational studies.

Method: From the 2010 OMOP Experiment, we sought drug-outcome-method combinations (DOMCs) that met consensus design criteria, yet repeatedly produced results contrary to expectation. We set aside DOMCs for which we could not agree on the suitability of the designs, then selected for an in-depth scrutiny one drug-outcome pair analyzed by a seemingly plausible methodological approach, whose results consistently disagreed with the a priori expectation.

Results: The OMOP "all-by-all" assessment of possible DOMCs yielded many combinations that would not be chosen by researchers as actual study options. Among those that passed a first level of scrutiny, two of seven drug-outcome pairs for which there were plausible research designs had anomalous results. The use of benzodiazepines was unexpectedly associated with acute renal failure and upper gastrointestinal bleeding. We chose the latter as an example for in-depth study. The factitious appearance of a bleeding risk may have been partly driven by an excess of procedures on the first day of treatment. A risk window definition that excluded the first day largely removed the spurious association.

Keywords: electronic health records; insurance claim data; medical product safety; monitoring; pharmacoepidemiology.

MeSH terms

Drug-Related Side Effects and Adverse Reactions / diagnosis
Humans
Observational Studies as Topic / methods*
Observational Studies as Topic / standards
Outcome Assessment, Health Care / methods*
Product Surveillance, Postmarketing / methods*
Reproducibility of Results
Research Design*