Development of case vignettes for assessment of the inter-rater variability of national validation teams for the point prevalence survey of healthcare-associated infections and antimicrobial use in European acute care hospitals

Abstract

Background: In 2016-17 the European Centre for Disease Prevention and Control (ECDC) organized the second point prevalence survey (PPS) of healthcare-associated infections (HCAIs) and antimicrobial use in European acute care hospitals. This survey included a validation study to maximize the accuracy of case identification and classification.

Aim: ECDC developed case vignettes to assess the performance of the national validation teams.

Methods: Case vignettes were developed by two medical doctors with experience in the management of HCAIs and antimicrobial stewardship. The case vignettes were based on actual clinical cases. The distribution of HCAIs among the case vignettes reflected the distribution of HCAIs in the previous PPS. All case vignettes were pilot-tested by three expert raters. Agreement among the expert raters was measured using kappa statistics.

Findings: Sixty case vignettes were developed. Twenty-nine of them were HCAI cases and 31 were cases without an HCAI. The inter-rater reliability using kappa statistics was 0.78 for the presence of HCAI and 0.89 for the antimicrobial use, respectively.

Conclusion: The agreement between the expert raters was very good for antimicrobial use and good for the presence of HCAI. Case vignettes can be a tool to support standardization of surveillance, improving the validity and comparability of the data.

Keywords: Case vignettes; Healthcare-associated infections; Point prevalence survey; Surveillance; Validity.