Analysis of risk of bias assessments in a sample of intervention systematic reviews, part I: many aspects of conduct and reporting need improvement

Abstract

Objectives: Current standards for systematic reviews (SRs) require adequate conduct and complete reporting of risk of bias (RoB) assessments of the individual studies included in the review. We investigated the conduct and reporting of RoB assessments reported in a sample of SRs of interventions for persons with cerebral palsy (CP).

Study design and setting: We included SRs published from 2014 to 2021. Authors worked in pairs to independently extract data on the characteristics of the SRs and to rate their conduct and reporting. The conduct of RoB assessment was appraised with the three AMSTAR-2 items related to RoB assessment. Reporting completeness was evaluated using the two items related to RoB assessment within studies in the PRISMA 2020 guidelines. We use descriptive statistics to report the consensus data, in accordance with our protocol.

Results: We included 145 SRs. Among the 128 (88.3%) SRs that assessed RoB, the standards for AMSTAR-2 item 9 (use of an adequate RoB tool) were partially or fully satisfied in 73 (57.0%). Across the 128 SRs that assessed RoB, 46 (35.9%) accounted for RoB in interpreting the SR's findings and, of the 49 that included a meta-analysis, 11 (22.4%) discussed the impact of RoB on this. 123 (96.1%) of the 128 SRs named the RoB tool that was used for at least one of the study designs they included, 96 (75.0%) specified the RoB items assessed and 89 (69.5%) reported the findings for each item, 81 (63.2%) fully reported the processes for RoB assessment, 68 (53.1%) reported how an overall RoB judgment was reached, and 74 (57.8%) reported an overall RoB assessment for every study.

Conclusion: The selection and application of RoB tools in this sample of SRs about interventions for CP are comparable to those reported in other recent studies. However, most SRs in this sample did not fully meet the appraisal standards of AMSTAR-2 regarding the adequacy of the RoB tool applied and other aspects of RoB assessment conduct; Cochrane SRs were a notable exception. Overall, reporting of RoB assessments was somewhat better than conduct, perhaps reflecting the more widespread uptake of the PRISMA guidelines. Our findings may be generalizable to some extent, considering the extensive literature reporting widespread inadequacies in health care-related intervention SRs and reports from other specialties that document similar RoB assessment deficiencies. As such, this study should remind authors, peer reviewers, and journal editors to follow the RoB assessment reporting guidelines of PRISMA 2020 and to understand the corresponding critical appraisal standards of AMSTAR-2. We recommend a shift of focus from the documentation of inadequate RoB assessments and well-known deficiencies in other components of SRs towards the implementation of changes to address these problems along with plans to evaluate their effectiveness.

Keywords: AMSTAR-2; Critical appraisal; Methodology; PRISMA 2020; Risk of bias; Systematic reviews.