VSR plus plus : Improving Visual Semantic Reasoning for Fine-Grained Image-Text Matching

Image-text matching has made great progresses recently, but there still remains challenges in fine-grained matching. To deal with this problem, we propose an Improved Visual Semantic Reasoning model (VSR++), which jointly models 1) global alignment between images and texts and 2) local correspondence between regions and words in a unified framework. To exploit their complementary advantages, we also develop a suitable learning strategy to balance their relative importance. As a result, our model can distinguish image regions and text words in a fine-grained level, and thus achieves the current state-of-the-art performance on two benchmark datasets.

Automated summary

The Improved Visual Semantic Reasoning model (VSR++) addresses the challenges in fine-grained image-text matching by jointly modeling global alignment and local correspondence. With a suitable learning strategy to balance their importance, the model achieves state-of-the-art performance on two benchmark datasets by distinguishing image regions and text words at a fine-grained level.