NeuroSense: Short-term emotion recognition and understanding based on spiking neural network modelling of spatio-temporal EEG patterns

Emotion recognition still poses a challenge lying at the core of the rapidly growing area of affective com-puting and is crucial for establishing a successful human-computer interaction. Identification and under-standing of emotions are achieved through various measures, such as subjective self-reports, face-tracking, voice analysis, gaze-tracking, as well as the analysis of autonomic and central neurophysiolog-ical measurements. Current approaches to emotion recognition based on electroencephalography (EEG) mostly rely on various handcrafted features extracted over relatively long time windows of EEG during participants exposure to appropriate affective stimuli. In this paper, we present a short-term emotion recognition framework based on spiking neural network (SNN) modelling of spatio-temporal EEG pat-terns. Our method relies on EEG signal segmentation based on detection of short-term changes in facial landmarks, and as such includes no computation of handcrafted EEG features. Differences between par-ticipants' EEG properties are taken into account via subject-dependent spike encoding in the formulated subject-independent emotion recognition task. We test our methods on the publicly available DEAP and MAHNOB-HCI databases due to the availability of both EEG and frontal face video data. Through an exhaustive hyperparameter optimisation strategy, we show that the proposed SNN-based representation of EEG spiking patterns provides valuable information for short-term emotion recognition. The obtained accuracies are 78.97% and 79.39% in arousal classification, and 67.76% and 72.12% in valence classifica-tion, on the DEAP and MAHNOB-HCI datasets, respectively. Furthermore, through the application of a brain-inspired SNN model, this study provides novel insight and helps in the understanding of the neural mechanisms involved in emotional processing in the context of audiovisual stimuli, such as affective videos. The presented results encourage the use of the proposed EEG processing methodology as a com-plement to existing features and methods commonly used for EEG-based emotion recognition, especially for short-term arousal recognition. (c) 2021 Elsevier B.V. All rights reserved.

Automated summary

Emotion recognition is a key challenge in the field of affective computing, and this study proposes a short-term emotion recognition framework based on spiking neural network without handcrafted EEG features. Testing on the DEAP and MAHNOB-HCI databases shows that the SNN-based EEG spiking patterns offer valuable information for short-term emotion recognition.