期刊
TRANSLATIONAL VISION SCIENCE & TECHNOLOGY
卷 5, 期 5, 页码 -出版社
ASSOC RESEARCH VISION OPHTHALMOLOGY INC
DOI: 10.1167/tvst.5.5.6
关键词
retina; image analysis; crowdsourcing
资金
- Fight for Sight (London)
- Special Trustees of Moorfields Eye Hospital and NIHR Biomedical Research Centre at Moorfields Eye Hospital
- UCL Institute of Ophthalmology
- Medical Research Council
- Cancer Research UK
- Research into Ageing
- Medical Research Council [G0401527, MR/N003284/1, G1000143] Funding Source: researchfish
- National Institute for Health Research [NF-SI-0512-10114] Funding Source: researchfish
- MRC [MR/N003284/1] Funding Source: UKRI
Purpose: Crowdsourcing is based on outsourcing computationally intensive tasks to numerous individuals in the online community who have no formal training. Our aim was to develop a novel online tool designed to facilitate large-scale annotation of digital retinal images, and to assess the accuracy of crowdsource grading using this tool, comparing it to expert classification. Methods: We used 100 retinal fundus photograph images with predetermined disease criteria selected by two experts from a large cohort study. The Amazon Mechanical Turk Web platform was used to drive traffic to our site so anonymous workers could perform a classification and annotation task of the fundus photographs in our dataset after a short training exercise. Three groups were assessed: masters only, nonmasters only and nonmasters with compulsory training. We calculated the sensitivity, specificity, and area under the curve (AUC) of receiver operating characteristic (ROC) plots for all classifications compared to expert grading, and used the Dice coefficient and consensus threshold to assess annotation accuracy. Results: In total, we received 5389 annotations for 84 images (excluding 16 training images) in 2 weeks. A specificity and sensitivity of 71% (95% confidence interval [CI], 69%-74%) and 87% (95% Cl, 86%-88%) was achieved for all classifications. The AUC in this study for all classifications combined was 0.93 (95% Cl, 0.91-0.96). For image annotation, a maximal Dice coefficient (-0.6) was achieved with a consensus threshold of 0.25. Conclusions: This study supports the hypothesis that annotation of abnormalities in retinal images by ophthalmologically naive individuals is comparable to expert annotation. The highest AUC and agreement with expert annotation was achieved in the nonmasters with compulsory training group. Translational Relevance: The use of crowdsourcing as a technique for retinal image analysis may be comparable to expert graders and has the potential to deliver timely, accurate, and cost-effective image analysis.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据