SSA-ICL: Multi-domain adaptive attention with intra-dataset continual learning for Facial expression recognition

Facial expression recognition (FER) is a kind of affective computing that identifies the emotional state represented in facial photographs. Various methods have been developed for completing this critical task. In spite of this progress, three significant obstacles, the interaction between spatial action units, the inadequacy of semantic information about spectral expressions and the unbalanced data distribution, are not well addressed. In this work, we propose SSA-ICL, a novel approach for FER, and solve these three difficulties inside a coherent framework. To address the first two challenges, we develop a Spectral and Spatial Attention (SSA) module that integrates spectral semantics with spatial locations to improve the performance of the model. We provide an Intra-dataset Continual Learning (ICL) module to combat the issue of long-tail distribution in FER datasets. By subdividing a single long-tail dataset into multiple sub-datasets, ICL repeatedly trains well-balanced representations from each subset and finally develop a independent classifier. We performed extensive experiments on two publicly available datasets, AffectNet and RAFDB. In comparison to existing attention modules, our SSA achieves an accuracy improvement of 3.8% similar to 6.7%, as evidenced by testing results. In the meanwhile, our proposed SSA-ICL can achieve superior or comparable performance to state-of-the-art FER methods (65.78% on AffectNet and 89.44% on RAFDB).(c) 2022 Elsevier Ltd. All rights reserved.

Automated summary

Facial expression recognition (FER) is a method of identifying emotions in facial photographs. Despite progress, three major challenges remain unaddressed: the interaction between spatial action units, inadequate semantic information about spectral expressions, and unbalanced data distribution. In this work, we propose SSA-ICL, a novel approach that solves these challenges by integrating spectral semantics with spatial locations and using an Intra-dataset Continual Learning (ICL) module to address data distribution.