Multi-level Alignment Network for Domain Adaptive Cross-modal Retrieval

Cross-modal retrieval is an important but challenging research task in the multimedia community. Most existing works of this task are supervised, which typically train models on a large number of aligned image-text/video-text pairs, making an assumption that training and testing data are drawn from the same distribution. If this assumption does not hold, traditional cross-modal retrieval methods may experience a performance drop at the evaluation. In this paper, we introduce a new task named as domain adaptive cross-modal retrieval, where training (source) data and testing (target) data are from different domains. The task is challenging, as there are not only the semantic gap and modality gap between visual and textual items, but also domain gap between source and target domains. Therefore, we propose a Multi-level Alignment Network (MAN) that has two mapping modules to project visual and textual modalities in a common space respectively, and three alignments are used to learn more discriminative features in the space. A semantic alignment is used to reduce the semantic gap, a cross-modality alignment and a cross-domain alignment are employed to alleviate the modality gap and domain gap. Extensive experiments in the context of domain-adaptive image-text retrieval and video-text retrieval demonstrate that our proposed model, MAN, consistently outperforms multiple baselines, showing a superior generalization ability for target data. Moreover, MAN establishes a new state-of-the-art for the large-scale text-to video retrieval on TRECVID 2017, 2018 Ad-hoc Video Search benchmark. (c) 2021 Elsevier B.V. All rights reserved.

Automated summary

This paper introduces a new task of domain adaptive cross-modal retrieval, addressing the scenario where training and testing data come from different domains. By proposing a Multi-level Alignment Network (MAN), the semantic, modality, and domain gaps are effectively reduced, enhancing the generalization ability for target data. Experiments show that MAN outperforms multiple baselines and achieves a new state-of-the-art in large-scale text-to-video retrieval.