Journal
MATHEMATICS
Volume 10, Issue 15, Pages -Publisher
MDPI
DOI: 10.3390/math10152644
Keywords
cross-modal retrieval; image-text retrieval; cross-modal similarity preserving; hashing algorithm; unsupervised learning
Categories
Funding
- National Natural Science Foundation of China [61841602]
- Natural Science Foundation of Shandong Province of China [ZR2018PF005, ZR2021MF017]
- Youth Innovation Science and Technology Team Foundation of Shandong Higher School [2021KJ031]
- Fundamental Research Funds for the Central Universities, JLU [93K172021K12]
Ask authors/readers for more resources
This paper introduces the task of image-text cross-modal retrieval and the proposed DRNPH method, which achieves cross-modal retrieval in the Hamming space, with constraints for consistent binary codes of similar sample pairs and minimal Hamming distances. Experimental results show that this method outperforms existing methods in various image-text retrieval scenarios.
The image-text cross-modal retrieval task, which aims to retrieve the relevant image from text and vice versa, is now attracting widespread attention. To quickly respond to the large-scale task, we propose an Unsupervised Deep Relative Neighbor Relationship Preserving Cross-Modal Hashing (DRNPH) to achieve cross-modal retrieval in the common Hamming space, which has the advantages of storage and efficiency. To fulfill the nearest neighbor search in the Hamming space, we demand to reconstruct both the original intra- and inter-modal neighbor matrix according to the binary feature vectors. Thus, we can compute the neighbor relationship among different modal samples directly based on the Hamming distances. Furthermore, the cross-modal pair-wise similarity preserving constraint requires the similar sample pair have an identical Hamming distance to the anchor. Therefore, the similar sample pairs own the same binary code, and they have minimal Hamming distances. Unfortunately, the pair-wise similarity preserving constraint may lead to an imbalanced code problem. Therefore, we propose the cross-modal triplet relative similarity preserving constraint, which demands the Hamming distances of similar pairs should be less than those of dissimilar pairs to distinguish the samples' ranking orders in the retrieval results. Moreover, a large similarity marginal can boost the algorithm's noise robustness. We conduct the cross-modal retrieval comparative experiments and ablation study on two public datasets, MIRFlickr and NUS-WIDE, respectively. The experimental results show that DRNPH outperforms the state-of-the-art approaches in various image-text retrieval scenarios, and all three proposed constraints are necessary and effective for boosting cross-modal retrieval performance.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available