期刊
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
卷 13, 期 3, 页码 1501-1513出版社
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TAFFC.2020.3008456
关键词
Facial animation; Hidden Markov models; Adaptation models; Speech recognition; Feature extraction; Data models; Speech driven facial animation; affective facial animation; domain adaptation; transfer learning
资金
- Scientific and Technological Research Council of Turkey (TUBITAK) [217E107]
This article improves affective facial animations through domain adaptation and data augmentation. The proposed models show significant MSE loss improvements in experiments, and the resulting facial animations are preferred by subjects in subjective evaluations.
Although speech driven facial animation has been studied extensively in the literature, works focusing on the affective content of the speech are limited. This is mostly due to the scarcity of affective audio-visual data. In this article, we improve the affective facial animation using domain adaptation by partially reducing the data scarcity. We first define a domain adaptation to map affective and neutral speech representations to a common latent space in which cross-domain bias is smaller. Then the domain adaptation is used to augment affective representations for each emotion category, including angry, disgust, fear, happy, sad, surprise, and neutral, so that we can better train emotion-dependent deep audio-to-visual (A2V) mapping models. Based on the emotion-dependent deep A2V models, the proposed affective facial synthesis system is realized in two stages: first, speech emotion recognition extracts soft emotion category likelihoods for the utterances; then a soft fusion of the emotion-dependent A2V mapping outputs form the affective facial synthesis. Experimental evaluations are performed on the SAVEE audio-visual dataset. The proposed models are assessed with objective and subjective evaluations. The proposed affective A2V system achieves significant MSE loss improvements in comparison to the recent literature. Furthermore, the resulting facial animations of the proposed system are preferred over the baseline animations in the subjective evaluations.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据