AOBERT: All-modalities-in-One BERT for multimodal sentiment analysis

Summary: The wide application of smart devices has enabled the use of multimodal data, but training networks with cross-modal information is still challenging due to modality gap. Additionally, the learning of inter-sample and inter-class relationships is often neglected. To address these issues, we propose HyCon, a framework for hybrid contrastive learning, which can explore cross-modal interactions, learn inter-sample and inter-class relationships, and reduce the modality gap. Our method outperforms baselines on multimodal sentiment analysis and emotion recognition.