Learning joint relationship attention network for image captioning

Image captioning aims at automatically describing the main content of an image with a complete and natural sentence. Existing attention-based methods often focus on visual features individually, while ignoring relationship information between image features that provides important guidance for generating captions. To alleviate this issue, in this work we propose the new Joint Relationship Attention Network (JRAN) that novelly explores the relationships between the features in the image. Technically, JRAN capitalizes on semantic features as s supplementary to the region features, fully learn two types of relationships, the visual relationships between region features and the visual-semantic relationships between region features and semantic features. Then, JRAN further make a dynamic trade-off between them during outputting the relationship representation. Moreover, we devise a new feature fusion based attention, which can adaptively fuse the region features and previously obtained relationship representation when generating different words. Extensive experiments on MSCOCO and Flickr30k, Flickr8k datasets show the superiority of our JRAN method qualitatively and quantitatively compared with several related state-of-the-art methods. More remarkably, JRAN achieves 28.3% and 58.2% on Flickr30k, and 22.7% and 55.3% on Flickr8k for the metrics BLEU4 and CIDEr.

Automated summary

This paper proposes a novel method for image captioning that generates complete and natural sentence descriptions of image content by exploring the relationships between image features. Experimental results demonstrate the superiority of this method compared to existing approaches both qualitatively and quantitatively.