M3: MultiModal Masking applied to sentiment analysis

A common issue when training multimodal architectures is that not all modalities contribute equally to the model's prediction and the network tends to over-rely on the strongest modality. In this work, we present M-3, a training procedure based on modality masking for deep multimodal architectures. During network training, we randomly select onemodality and mask its features, forcing the model to make its prediction in the absence of this modality. This structured regularization allows the network to better exploit complementary information in input modalities. We implement M-3 as a generic layer that can be integrated with any multimodal architecture. Our experiments show that M-3 outperforms other masking schemes and improves performance for our strong baseline. We evaluate M-3 for multimodal sentiment analysis on CMU-MOSEI, achieving results comparable to the state-of-the-art.

Automated summary

In this work, a training procedure named M-3 is proposed for deep multimodal architectures, which involves masking features of one modality during network training to force the model to make predictions in the absence of that modality. The structured regularization of M-3 allows the network to better leverage complementary information from input modalities. Experimental results show that M-3 outperforms other masking schemes and achieves performance improvements for multimodal sentiment analysis tasks.