Multi-modal fusion network with complementarity and importance for emotion recognition

Multimodal emotion recognition, that is, emotion recognition uses machine learning to generate multi-modal features on the basis of videos which has become a research hotspot in the field of artificial intelligence. Traditional multi-modal emotion recognition method only simply connects multiple modalities, and the interactive utilization rate of modal information is low, and it cannot reflect the real emotion under the conflict of modal fea-tures well. This article first proves that effective weighting can improve the discrimination between modalities. Therefore, this paper takes into account the importance differences between multiple modalities, and assigns weights to them through the importance atten-tion network. At the same time, considering that there is a certain complementary relation-ship between the modalities, this paper constructs an attention network with complementary modalities. Finally, the reconstructed features are fused to obtain a multi-modal feature with good interaction. The method proposed in this paper is compared with traditional methods in public datasets. The test results show that our method is accu-rate in It performs well in both the rate and confusion matrix metrics.(c) 2022 Elsevier Inc. All rights reserved.

Automated summary

Multimodal emotion recognition, using machine learning to generate multi-modal features from videos, is a research hotspot in the field of artificial intelligence. This paper improves the discrimination between modalities through effective weighting and constructs an attention network to capture the complementary relationship between modalities, resulting in a multi-modal feature with good interaction.