Fusing Frequency-Domain Features and Brain Connectivity Features for Cross-Subject Emotion Recognition

Frequency-domain (FD) features reveal the activated patterns of individual local brain regions responding to different emotions, whereas brain connectivity (BC) features involve the coordination of multiple brain regions for generating emotional responses; these two types of features are complementary to each other. To date, the fusion of these two types of features for electroencephalography (EEG)-based cross-subject emotion recognition remains to be fully investigated due to the intersubject variability in EEG signals. In this article, we first attempt to investigate these fused features for cross-subject emotion recognition from multiple perspectives, including critical frequency bands, complementary characteristics for each emotional state, critical channels, and crucial connections, using a fast and robust approximate empirical kernel map-fusion-based support vector machine (AEKM-Fusion-SVM) method. The experimental results on the SJTU emotion EEG dataset (SEED), BCI2020-A, and BCI2020-B datasets reveal that: 1) the AEKM-fusion method improves the effectiveness and efficiency of the fusion of features of different dimensions; 2) the recognition accuracy of the fused features outperforms each individual feature, and this outperformance is more significant in the high-frequency bands (i.e., the beta and gamma bands); 3) the fused features significantly enhance the classification performance for negative emotion; and 4) the fused features built with 27 selected channels achieve comparable performance to that of the fused features built with the full number of channels (i.e., 62 channels), allowing for easier establishment of brain-computer interface (BCI) systems in real-world scenarios. Our study enriches the research of emotion-related brain mechanisms and provides new insight into affective computing.

Automated summary

This article investigates the fusion of frequency-domain and brain connectivity features for cross-subject emotion recognition using electroencephalography (EEG). Through multiple perspectives, including critical frequency bands, complementary characteristics for each emotional state, critical channels, and crucial connections, the study reveals that the fused features outperform individual features, especially in high-frequency bands, and significantly enhance classification performance for negative emotion. The findings contribute to the research on emotion-related brain mechanisms and offer new insights into affective computing.