3.8 Proceedings Paper

Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching

出版社

IEEE COMPUTER SOC
DOI: 10.1109/WACV51458.2022.00254

关键词

-

资金

  1. CERCA Programme/Generalitat de Catalunya
  2. AGAUR [2019PROD00090, 2019FIB01233]
  3. UAB [B18P0073]
  4. [PID2020-116298GB-I00]

向作者/读者索取更多资源

The existing datasets for image-text matching task lack the ability to accurately measure semantic relevance. This study proposes two metrics to evaluate the semantic relevance of image-text pairs and introduces a new strategy to improve model performance. Experiments show significant improvements in scenarios with limited training data.
The task of image-text matching aims to map representations from different modalities into a common joint visual-textual embedding. However, the most widely used datasets for this task, MSCOCO and Flickr30K, are actually image captioning datasets that offer a very limited set of relationships between images and sentences in their ground-truth annotations. This limited ground truth information forces us to use evaluation metrics based on binary relevance: given a sentence query we consider only one image as relevant. However; many other relevant images or captions may be present in the dataset. In this work, we propose two metrics that evaluate the degree of semantic relevance of retrieved items, independently of their annotated binary relevance. Additionally, we incorporate a novel strategy that uses an image captioning metric, CIDEr; to define a Semantic Adaptive Margin (SAM) to be optimized in a standard triplet loss. By incorporating our formulation to existing models, a large improvement is obtained in scenarios where available training data is limited. We also demonstrate that the performance on the annotated image-caption pairs is maintained while improving on other non-annotated relevant items when employing the All training set. The code for our new metric can be found at github.com/furkanbiten/ncs metric and the model implementation at github.com/andrespmd/semantic adaptive margin.

作者

我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。

评论

主要评分

3.8
评分不足

次要评分

新颖性
-
重要性
-
科学严谨性
-
评价这篇论文

推荐

暂无数据
暂无数据