Looking Under the Hood: Deep Neural Network Visualization to Interpret Whole-Slide Image Analysis Outcomes for Colorectal Polyps

Histopathological characterization of colorectal polyps is an important principle for determining the risk of colorectal cancer and future rates of surveillance for patients. The process of characterization is time-intensive and requires years of specialized medical training. In this work, we propose a deep-learning-based image analysis approach that not only can accurately classify different types of polyps in whole-slide images, but also generates major regions and features on the slide through a model visualization approach. We argue that this visualization approach will make sense of the underlying reasons for the classification outcomes, significantly reduce the cognitive burden on clinicians, and improve the diagnostic accuracy for whole-slide image characterization tasks. Our results show the efficacy of this network visualization approach in recovering decisive regions and features for different types of polyps on whole-slide images according to the domain expert pathologists.