Distributed Human Intelligence for Colonic Polyp Classification in Computer-aided Detection for CT Colonography

Purpose: To assess the diagnostic performance of distributed human intelligence for the classification of polyp candidates identified with computer-aided detection (CAD) for computed tomographic (CT) colonography. Materials and Methods: This study was approved by the institutional Office of Human Subjects Research. The requirement for informed consent was waived for this HIPAA-compliant study. CT images from 24 patients, each with at least one polyp of 6 mm or larger, were analyzed by using CAD software to identify 268 polyp candidates. Twenty knowledge workers (KWs) from a crowdsourcing platform labeled each polyp candidate as a true or false polyp. Two trials involving 228 KWs were conducted to assess reproducibility. Performance was assessed by comparing the area under the receiver operating characteristic curve (AUC) of KWs with the AUC of CAD for polyp classification. Results: The detection-level AUC for KWs was 0.845 +/- 0.045 (standard error) in trial 1 and 0.855 +/- 0.044 in trial 2. These were not significantly different from the AUC for CAD, which was 0.859 +/- 0.043. When polyp candidates were stratified by difficulty, KWs performed better than CAD on easy detections; AUCs were 0.951 +/- 0.032 in trial 1, 0.966 +/- 0.027 in trial 2, and 0.877 +/- 0.048 for CAD (P = .039 for trial 2). KWs who participated in both trials showed a significant improvement in performance going from trial 1 to trial 2; AUCs were 0.759 +/- 0.052 in trial 1 and 0.839 +/- 0.046 in trial 2 (P = .041). Conclusion: The performance of distributed human intelligence is not significantly different from that of CAD for colonic polyp classification.