Journal
IEEE TRANSACTIONS ON IMAGE PROCESSING
Volume 30, Issue -, Pages 9193-9207Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TIP.2021.3123553
Keywords
Learning systems; Feature extraction; Visualization; Semantics; Correlation; Benchmark testing; Transformers; Image-text retrieval; memory network; attention mechanism; transformer
Funding
- National Key Research and Development Program of China [2018AAA0100704]
- NSF of China [62076162]
- Shanghai Municipal Science and Technology Major Project [2021SHZDZX0102]
- Shanghai Municipal Science and Technology Key Project [20511100300]
Ask authors/readers for more resources
The MEMBER method introduces global memory banks to enable fine-grained alignment and fusion between images and texts in embedding learning paradigm, achieving mutual embedding enhancement and maintaining retrieval efficiency. Extensive experiments show that MEMBER outperforms state-of-the-art approaches on two large-scale benchmark datasets.
Image-text retrieval aims to capture the semantic correlation between images and texts. Existing image-text retrieval methods can be roughly categorized into embedding learning paradigm and pair-wise learning paradigm. The former paradigm fails to capture the fine-grained correspondence between images and texts. The latter paradigm achieves fine-grained alignment between regions and words, but the high cost of pair-wise computation leads to slow retrieval speed. In this paper, we propose a novel method named MEMBER by using Memory-based EMBedding Enhancement for image-text Retrieval (MEMBER), which introduces global memory banks to enable fine-grained alignment and fusion in embedding learning paradigm. Specifically, we enrich image (resp., text) features with relevant text (resp., image) features stored in the text (resp., image) memory bank. In this way, our model not only accomplishes mutual embedding enhancement across two modalities, but also maintains the retrieval efficiency. Extensive experiments demonstrate that our MEMBER remarkably outperforms state-of-the-art approaches on two large-scale benchmark datasets.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available