Rationale and objectives: Rank-order experiments often provide a reasonable method of determining whether a large-scale receiver operating characteristic study can be justified. The authors' purpose was to formalize a proposed method for analyzing rank-order imaging experiments and provide methods that can be used in determining sample sizes for both cases and raters.
Materials and methods: Simulations were conducted to determine the adequacy of the normal approximation of a statistic used to test the null hypothesis of random ordering. For a multireader experiment, formulas are presented and guidelines are provided to enable investigators to determine the number of required readers (raters) and cases for a specific study.
Results: When there are at least five ordered images per case, 10 cases are sufficient to test a random rank order. When there are only three or four images for a case, 20 cases are required. The authors constructed tables of statistical power for selected numbers of ordered images, numbers of cases, and degrees of trend, and they also provide an approximation for use in situations that are not tabled.
Conclusion: The statistical methods for analyzing rank-order experiments and estimating sample sizes for study planning are relatively simple to implement. The derived formulas for sample size estimation, when applied to typical imaging experiments, indicate that modest numbers of cases and readers are required for rank-order studies.