Automatic Disease Annotation From Radiology Reports Using Artificial Intelligence Implemented by a Recurrent Neural Network

OBJECTIVE. Radiology reports are rich resources for biomedical researchers. Before utilization of radiology reports, experts must manually review these reports to identify the categories. In fact, automatically categorizing electronic medical record (EMR) text with key annotation is difficult because it has a free-text format. To address these problems, we developed an automated system for disease annotation. MATERIALS AND METHODS. Reports of musculoskeletal radiography examinations performed from January 1, 2016, through December 31, 2016, were exported from the database of Hanyang University Medical Center. After sentences not written in English and sentences containing typos were excluded, 3032 sentences were included. We built a system that uses a recurrent neural network (RNN) to automatically identify fracture and nonfracture cases as a preliminary study. We trained and tested the system using orthopedic surgeon-classified reports. We evaluated the system for the number of layers in the following two ways: the word error rate of the output sentences and performance as a binary classifier using standard evaluation metrics including accuracy, precision, recall, and F1 score. RESULTS. The word error rate using Levenshtein distance showed the best performance in the three-layer model at 1.03%. The three-layer model also showed the highest overall performance with the highest precision (0.967), recall (0.967), accuracy (0.982), and F1 score (0.967). CONCLUSION. Our results indicate that the RNN-based system has the ability to classify important findings in radiology reports with a high F1 score. We expect that our system can be used in cohort construction such as for retrospective studies because it is efficient for analyzing a large amount of data.