Visual Interpretation of CNN Prediction Through Layerwise Sequential Selection of Discernible Neurons

In recent years, researchers have been working to interpret the insights of deep neural networks in the pursuit of overcoming their opaqueness and so-called 'black-box' tag. In this work, we present a new posthoc visual interpretation technique that finds out discriminative image regions contributing highly towards networks' prediction. We select most discernible set of neurons per layer and engineer the forward pass operation to gradually reach most discriminative image locations. While searching for discernible neurons, existing approaches either end up with low-resolution visualization maps, or suffer lack of neuron discriminativity in the way. Moreover, some methods concentrate only on current layer information overlooking meaningful information from adjacent layers, limiting the overall scope of selection. We address these issues by exploring the layer-to-layer connected structure of a neuron and obtaining contributions from its current layer along with its adjacent layers, e.g., succeeding and preceding layer. We introduce a score function where such contributions are assembled with appropriate priorities. Layerwise discernible neurons are then selected based on top scores, ensuring a reliable and credible selection. We validate our proposal through objective and subjective evaluations by examining its performance in terms of models' faithfulness and human-trust, where we visualize its efficacy over other existing methods. We also perform sets of sanity check experiments on our method to show its overall reliability as a visualization map.

Automated summary

In this study, a new posthoc visual interpretation technique is proposed to identify discriminative image regions contributing highly towards networks' prediction. By exploring the layer-to-layer connected structure of neurons and obtaining contributions from adjacent layers, reliable and credible discernible neurons per layer are selected.