Modeling Hierarchical Uncertainty for Multimodal Emotion Recognition in Conversation

Approximating the uncertainty of an emotional AI agent is crucial for improving the reliability of such agents and facilitating human-in-the-loop solutions, especially in critical scenarios. However, none of the existing systems for emotion recognition in conversation (ERC) has attempted to estimate the uncertainty of their predictions. In this article, we present HU-Dialogue, which models hierarchical uncertainty for the ERC task. We perturb contextual attention weight values with source-adaptive noises within each modality, as a regularization scheme to model context-level uncertainty and adapt the Bayesian deep learning method to the capsule-based prediction layer to model modality-level uncertainty. Furthermore, a weight-sharing triplet structure with conditional layer normalization is introduced to detect both invariance and equivariance among modalities for ERC. We provide a detailed empirical analysis for extensive experiments, which shows that our model outperforms previous state-of-the-art methods on three popular multimodal ERC datasets.

Automated summary

Approximating the uncertainty of an emotional AI agent is crucial for improving reliability and facilitating human-in-the-loop solutions. In this article, HU-Dialogue is presented, a model that incorporates hierarchical uncertainty for emotion recognition in conversation (ERC) task. Experimental results show that our model outperforms previous state-of-the-art methods on popular multimodal ERC datasets.