Emotion recognition framework using multiple modalities for an effective human-computer interaction

Human emotions are subjective reactions to objects or events that are related to diverse physiological, behavioral and intellectual changes. The research community is gaining more interest in emotion recognition due to its vast applications including human-computer interaction, virtual reality, self-driving, digital content entertainment, human behavior monitoring, and medicine. Electroencephalogram (EEG) signals that are collected from the brain are playing a massive part in the advancement of brain-computer interface systems. The current techniques that are using EEG signals for emotion recognition are lacking in subject-independent or cross-subject emotion analysis. Additionally, there is a lack of multimodal approaches that combine EEG data with other modalities. In view of the stated deficiencies, this study presents an efficient multimodal strategy for cross-subject emotion recognition utilizing EEG and facial gestures. The proposed method fuses the spectral and statistical features extracted from the EEG data with a histogram of oriented gradients and local binary patterns features extracted from the facial images. Following on, support vector machines, k-nearest neighbor, and ensemble are employed for emotion classification. Additionally, the class misbalance problem is solved using the up-sampling approach. The accuracy of the suggested method is assessed on the dataset of emotion analysis using physiological signals with tenfold cross-validation. The findings of the research study are promising, with the highest accuracy of 97.25% for valence and 96.1% for arousal, respectively.

Automated summary

Human emotions are subjective reactions to various changes in physiology, behavior, and intellect. The research community is increasingly interested in emotion recognition due to its wide range of applications such as human-computer interaction, virtual reality, self-driving, and medicine. This study proposes an efficient multimodal strategy for cross-subject emotion recognition using EEG and facial gestures, combining spectral and statistical features from EEG data with histogram of oriented gradients and local binary patterns features from facial images. Support vector machines, k-nearest neighbor, and ensemble classifiers are utilized for emotion classification, with the issue of class imbalance being addressed using the up-sampling approach. The proposed method achieves promising results with high accuracy rates of 97.25% for valence and 96.1% for arousal, respectively.