期刊
IEEE TRANSACTIONS ON CYBERNETICS
卷 49, 期 3, 页码 1110-1122出版社
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TCYB.2018.2797176
关键词
Affective brain-computer interactions; deep learning; EEG; emotion recognition; eye movements; multimodal deep neural networks
类别
资金
- National Key Research and Development Program of China [2017YFB1002501]
- National Natural Science Foundation of China [61673266]
- Major Basic Research Program of Shanghai Science and Technology Committee [15JC1400103]
- ZBYY-MOE Joint Funding [6141A02022604]
- Technology Research and Development Program of China Railway Corporation [2016Z003-B]
- Fundamental Research Funds for the Central Universities
- Ministry of Education and Science of the Russian Federation [14.756.31.0001]
- Polish National Science Center [2016/20/W/N24/00354]
In this paper, we present a multimodal emotion recognition framework called EmotionMeter that combines brain waves and eye movements. To increase the feasibility and wearability of EmotionMeter in real-world applications, we design a six-electrode placement above the ears to collect electroencephalography (EEG) signals. We combine EEG and eye movements for integrating the internal cognitive states and external subconscious behaviors of users to improve the recognition accuracy of EmotionMeter. The experimental results demonstrate that modality fusion with multimodal deep neural networks can significantly enhance the performance compared with a single modality, and the best mean accuracy of 85.11% is achieved for four emotions (happy, sad, fear, and neutral). We explore the complementary characteristics of EEG and eye movements for their representational capacities and identify that EEG has the advantage of classifying happy emotion, whereas eye movements outperform EEG in recognizing fear emotion. To investigate the stability of EmotionMeter over time, each subject performs the experiments three times on different days. EmotionMeter obtains a mean recognition accuracy of 72.39% across sessions with the six-electrode EEG and eye movement features. These experimental results demonstrate the effectiveness of EmotionMeter within and between sessions.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据