Comparison of non-Cochrane systematic reviews and their published protocols: differences occurred frequently but were seldom explained

Abstract

Objective: To quantify the prevalence of differences in the reported methods between non-Cochrane systematic reviews (SRs) and their protocols and the extent to which these were reported and explained.

Study design and setting: We searched MEDLINE and Embase to identify protocols of non-Cochrane SRs published in 2012 and 2013. Using various methods, we searched for their corresponding SRs up to December 2016. The SRs and protocols were compared with respect to the methods-related "Preferred Reporting Items for Systematic review and Meta-Analysis Protocols" (PRISMA-P).

Results: We included 80 SRs and their protocols. Almost all SRs (92.5%) differed from their protocols in at least one of the methods-related PRISMA-P items (no. 7-17) and their subcategories. Half the SRs (48.8%) had a major difference in at least one item. On average, each SR differed from its protocol in 3.2 items, of which one comprised a major difference. Only 10% of all differences were reported in the SR, two-thirds with an explanation (7.0% in total).

Conclusion: The reporting quality and transparency of non-Cochrane SRs requires further improvement. Authors should report and explain all important changes made to the protocol in the SR publication. The updated PRISMA statement should include guidance regarding this matter.

Keywords: Methodology; PRISMA; Protocol; Reporting quality; Systematic review; Transparency.