Efficiency and safety of automated label cleaning on multimodal retinal images

Tian Lin; Meng Wang; Aidi Lin; Xiaoting Mai; Huiyu Liang; Yih-Chung Tham; Haoyu Chen

doi:10.1038/s41746-024-01424-x

Efficiency and safety of automated label cleaning on multimodal retinal images

NPJ Digit Med. 2025 Jan 5;8(1):10. doi: 10.1038/s41746-024-01424-x.

Authors

Tian Lin^#^{1

2}, Meng Wang^#³, Aidi Lin^{1

2}, Xiaoting Mai^{1

2}, Huiyu Liang^{1

2}, Yih-Chung Tham^{4

5}, Haoyu Chen^{6

7}

Affiliations

¹ Joint Shantou International Eye Center, Shantou University and the Chinese University of Hong Kong, Shantou, Guangdong, 515041, China.
² Shantou University Medical College, Shantou, Guangdong, 515041, China.
³ Beth Israel Deaconess Medical Center, Harvard Medical School, 330 Brookline Ave, Boston, MA, 02215, USA.
⁴ Centre for Innovation & Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, 117549, Singapore.
⁵ Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, 117549, Singapore.
⁶ Joint Shantou International Eye Center, Shantou University and the Chinese University of Hong Kong, Shantou, Guangdong, 515041, China. drchenhaoyu@gmail.com.
⁷ Shantou University Medical College, Shantou, Guangdong, 515041, China. drchenhaoyu@gmail.com.

^# Contributed equally.

Abstract

Label noise is a common and important issue that would affect the model's performance in artificial intelligence. This study assessed the effectiveness and potential risks of automated label cleaning using an open-source framework, Cleanlab, in multi-category datasets of fundus photography and optical coherence tomography, with intentionally introduced label noise ranging from 0 to 70%. After six cycles of automatic cleaning, significant improvements are achieved in label accuracies (3.4-62.9%) and dataset quality scores (DQS, 5.1-74.4%). The majority (86.6 to 97.5%) of label errors were accurately modified, with minimal missed (0.5-2.8%) or misclassified (0.4-10.6%). The classification accuracy of RETFound significantly improved by 0.3-52.9% when trained with the datasets after cleaning. We also developed a DQS-guided cleaning strategy to mitigate over-cleaning. Furthermore, external validation on EyePACS and APTOS-2019 datasets boosted label accuracy by 1.3 and 1.8%, respectively. This approach automates label correction, enhances dataset reliability, and strengthens model performance efficiently and safely.