期刊
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
卷 10, 期 1, 页码 60-75出版社
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TAFFC.2017.2713783
关键词
Multimodal emotion recognition; classifier fusion; data fusion; convolutional neural networks
资金
- Estonian Research Grant [PUT638]
- European Commission
- Estonian Centre of Excellence in IT (EXCITE) - European Regional Development Fund
- European Network on Integrating Vision and Language (iV&L Net) ICT COST Action [IC1307]
- [TIN2013-43478-P]
- [TIN2016-74946-P]
This paper presents a multimodal emotion recognition system, which is based on the analysis of audio and visual cues. From the audio channel, Mel-Frequency Cepstral Coefficients, Filter Bank Energies and prosodic features are extracted. For the visual part, two strategies are considered. First, facial landmarks' geometric relations, i.e., distances and angles, are computed. Second, we summarize each emotional video into a reduced set of key-frames, which are taught to visually discriminate between the emotions. In order to do so, a convolutional neural network is applied to key-frames summarizing videos. Finally, confidence outputs of all the classifiers from all the modalities are used to define a new feature space to be learned for final emotion label prediction, in a late fusion/stacking fashion. The experiments conducted on the SAVEE, eNTERFACE'05, and RML databases show significant performance improvements by our proposed system in comparison to current alternatives, defining the current state-of-the-art in all three databases.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据