Objectives: The use of analytic rubrics in assessing and grading students' performance has become more prominent among instructors due to its reliability and validity in ensuring consistency in student evaluation. However, there is limited evidence demonstrating the consistency of examiner judgments between analytic marking rubrics and holistic marking rubrics.
Methods: Therefore, we aimed to compare the consistency of marks given using holistic marking methods and analytic rubrics at an Australian university by analyzing the mean mark differences and number of adjudications between two rubric types as well as the inter-rater reliability between two assessors. We analyzed all scores for project manuscripts between 2016 and 2021 for Honours medical students. We compared the mean mark differences graded using the Kruskal-Wallis test and Welch t-test. We used chi-squared tests to compare the frequency of adjudications for each rubric type. We assessed interrater reliability by comparing the marks between the two examiners utilizing Pearson correlation.
Results: We found that analytic rubrics have lower mean mark differences and fewer adjudicators are required. We showed a strong positive association between the consistency of marks given and the use of analytic rubrics when compared to holistic marking. Pearson correlation showed a low but stronger correlation between marks awarded by the two assessors when analytic rubrics were used (r = 0.36), compared to holistic marking rubrics (r = 0.24).
Conclusions: Our findings suggest that the use of analytic rubrics may increase the consistency and reliability between two independent examiners in marking medical students' work.
Keywords: adjudication; analytic rubrics; consistency; holistic marking; inter-rater reliability; mark differences; rubric.
© The Author(s) 2024.