Journal
IEEE TRANSACTIONS ON IMAGE PROCESSING
Volume 30, Issue -, Pages 4384-4394Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TIP.2021.3071688
Keywords
Training; Kernel; Interpolation; Data models; Geometry; Learning systems; Deep learning; Multi-modal learning; multi-view learning; cross-modal retrieval; nonlinear embeddings; supervised embeddings; RBF interpolators
Ask authors/readers for more resources
Numerous approaches exist in the literature for learning low-dimensional representations of multi-modal data collections, yet the generalizability of multi-modal nonlinear embeddings to unseen data has been overlooked. The study highlights the importance of the regularity of interpolation functions for successful generalization in multi-modal classification and retrieval problems, alongside criteria such as between-class separation and cross-modal alignment. The proposed multi-modal nonlinear representation learning algorithm, inspired by theoretical findings, shows promising performance in applications such as multi-modal image classification and cross-modal image-text retrieval.
While many approaches exist in the literature to learn low-dimensional representations for data collections in multiple modalities, the generalizability of multi-modal nonlinear embeddings to previously unseen data is a rather overlooked subject. In this work, we first present a theoretical analysis of learning multi-modal nonlinear embeddings in a supervised setting. Our performance bounds indicate that for successful generalization in multi-modal classification and retrieval problems, the regularity of the interpolation functions extending the embedding to the whole data space is as important as the between-class separation and cross-modal alignment criteria. We then propose a multi-modal nonlinear representation learning algorithm that is motivated by these theoretical findings, where the embeddings of the training samples are optimized jointly with the Lipschitz regularity of the interpolators. Experimental comparison to recent multi-modal and single-modal learning algorithms suggests that the proposed method yields promising performance in multi-modal image classification and cross-modal image-text retrieval applications.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available