Journal
COMPUTER VISION AND IMAGE UNDERSTANDING
Volume 173, Issue -, Pages 33-45Publisher
ACADEMIC PRESS INC ELSEVIER SCIENCE
DOI: 10.1016/j.cviu.2017.12.004
Keywords
Image Understanding; Commonsense Reasoning; Vision; Reasoning
Funding
- National Science Foundation [SMA 1540917, CNS 1544797]
Ask authors/readers for more resources
Two of the fundamental tasks in image understanding using text are caption generation and visual question answering (Antol et al., 2015; Xiong et al., 2016). This work presents an intermediate knowledge structure that can be used for both tasks to obtain increased interpretability. We call this knowledge structure Scene Description Graph (SDG), as it is a directed labeled graph, representing objects, actions, regions, as well as their attributes, along with inferred concepts and semantic (from KM-Ontology (Clark et al., 2004)), ontological (i.e. superclass, hasProperty), and spatial relations. Thereby a general architecture is proposed in which a system can represent both the content and underlying concepts of an image using an SDG. The architecture is implemented using generic visual recognition techniques and commonsense reasoning to extract graphs from images. The utility of the generated SDGs is demonstrated in the applications of image captioning, image retrieval, and through examples in visual question answering. The experiments in this work show that the extracted graphs capture syntactic and semantic content of images with reasonable accuracy.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available