ICDN: integrating consistency and difference networks by transformer for multimodal sentiment analysis

The sentiment of human language is usually reflected through multimodal forms such as natural language, facial expression, and voice intonation. However, the previous research methods uniformly treated different modalities of time series alignment and ignored the missing modal information fragments. The main challenge is the partial absence of multimodal information. In this work, the integrating consistency and difference networks(ICDN) is firstly proposed to model modalities interaction through mapping and generalization learning, which includes a special cross-modal Transformer designed to map other modalities to the target modality. Then, the unimodal sentiment labels are obtained through self-supervision to guide the final sentiment analysis. Compared with other popular multimodal sentiment analysis methods, we obtain better sentiment classification results on CMU-MOSI and CMU-MOSEI benchmark datasets.

Automated summary

In this paper, the authors propose an integrating consistency and difference networks (ICDN) method for multimodal sentiment analysis. The ICDN method models the interaction between different modalities through mapping and generalization learning, and uses a special cross-modal Transformer design to handle missing modal information. Unimodal sentiment labels are obtained through self-supervision to guide the sentiment analysis process. Experimental results demonstrate that the proposed method achieves better sentiment classification performance on the CMU-MOSI and CMU-MOSEI benchmark datasets.