Automated Early Detection of Diabetic Retinopathy

Purpose: To compare the performance of automated diabetic retinopathy (DR) detection, using the algorithm that won the 2009 Retinopathy Online Challenge Competition in 2009, the Challenge2009, against that of the one currently used in EyeCheck, a large computer-aided early DR detection project. Design: Evaluation of diagnostic test or technology. Participants: Fundus photographic sets, consisting of 2 fundus images from each eye, were evaluated from 16 670 patient visits of 16 670 people with diabetes who had not previously been diagnosed with DR. Methods: The fundus photographic set from each visit was analyzed by a single retinal expert; 793 of the 16 670 sets were classified as containing more than minimal DR (threshold for referral). The outcomes of the 2 algorithmic detectors were applied separately to the dataset and were compared by standard statistical measures. Main Outcome Measures: The area under the receiver operating characteristic curve (AUC), a measure of the sensitivity and specificity of DR detection. Results: Agreement was high, and examination results indicating more than minimal DR were detected with an AUC of 0.839 by the EyeCheck algorithm and an AUC of 0.821 for the Challenge2009 algorithm, a statistically nonsignificant difference (z-score, 1.91). If either of the algorithms detected DR in combination, the AUC for detection was 0.86, the same as the theoretically expected maximum. At 90% sensitivity, the specificity of the EyeCheck algorithm was 47.7% and that of the Challenge2009 algorithm was 43.6%. Conclusions: Diabetic retinopathy detection algorithms seem to be maturing, and further improvements in detection performance cannot be differentiated from best clinical practices, because the performance of competitive algorithm development now has reached the human intrareader variability limit. Additional validation studies on larger, well-defined, but more diverse populations of patients with diabetes are needed urgently, anticipating cost-effective early detection of DR in millions of people with diabetes to triage those patients who need further care at a time when they have early rather than advanced DR.