Journal
JOURNAL OF VISUAL COMMUNICATION AND IMAGE REPRESENTATION
Volume 78, Issue -, Pages -Publisher
ACADEMIC PRESS INC ELSEVIER SCIENCE
DOI: 10.1016/j.jvcir.2021.103138
Keywords
Image captioning; Encoder-decoder; Spatial information; Adaptive attention
Funding
- Fundamental Research Funds for the Central Universities of China [191010001]
- Hubei Key Laboratory of Transportation Internet of Things [2018IOT003, 2020III026GX]
- Ministry of Science and Technology, Taiwan [MOST 109-2634-F-007-013]
Ask authors/readers for more resources
The proposed image captioning scheme based on adaptive spatial information attention (ASIA) effectively extracts spatial information of salient objects, utilizes different techniques in encoding and decoding stages, improving captioning performance according to extensive experiments on two datasets.
Although attention mechanisms are exploited widely in encoder-decoder neural network-based image captioning framework, the relation between the selection of salient image regions and the supervision of spatial information on local and global representation learning was overlooked, thereby degrading captioning performance. Consequently, we propose an image captioning scheme based on adaptive spatial information attention (ASIA), extracting a sequence of spatial information of salient objects in a local image region or an entire image. Specifically, in the encoding stage, we extract the object-level visual features of salient objects and their spatial bounding-box. We obtain the global feature maps of an entire image, which are fused with local features and the fused features are fed into the LSTM-based language decoder. In the decoding stage, our adaptive attention mechanism dynamically selects the corresponding image regions specified by an image description. Extensive experiments conducted on two datasets demonstrate the effectiveness of the proposed method.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available