Mining Interpretable AOG Representations From Convolutional Networks via Active Question Answering

In this paper, we present a method to mine object-part patterns from cony-layers of a pre-trained convolutional neural network (CNN). The mined object-part patterns are organized by an And-Or graph (AOG). This interpretable AOG representation consists of a four-layer semantic hierarchy, i.e., semantic parts, part templates, latent patterns, and neural units. The AOG associates each object part with certain neural units in feature maps of cony-layers. The AOG is constructed with very few annotations (e.g., 3-20) of object parts. We develop a question-answering (QA) method that uses active human-computer communications to mine patterns from a pre-trained CNN, in order to explain features in cony-layers incrementally. During the learning process, our QA method uses the current AOG for part localization. The OA method actively identifies objects, whose feature maps cannot be explained by the AOG. Then, our method asks people to annotate parts on the unexplained objects, and uses answers to discover CNN patterns corresponding to newly labeled parts. In this way, our method gradually grows new branches and refines existing branches on the AOG to semanticize CNN representations. In experiments, our method exhibited a high learning efficiency. Our method used about 1/6-1/3 of the part annotations for training, but achieved similar or better part-localization performance than fast-RCNN methods.

Automated summary

This paper presents a method for mining object-part patterns from a pre-trained CNN, organizing the mined patterns using an And-Or graph to semanticize CNN representations. The method exhibits high learning efficiency in experiments and achieves similar or better part-localization performance compared to fast-RCNN methods.