Appearance and geometry transformer for facial expression recognition in the wild

Facial expression recognition (FER) in the wild is considered to be one of the most challenging tasks in computer vision because it requires identifying the categories of expressions presented by the human face under adverse conditions, such as dynamic poses, extreme illuminations, and partial occlusion. In this paper, we propose a model called the appearance and geometry transformer (AGT), which is a self-attention-based deep neural network, to improve the accuracy of recognition of FER in the wild. The AGT is designed to simultaneously perform feature extraction and fusion on heterogeneous data, including images and graphs, by using two transformer pathways. A pre-trained DeiT model is introduced to the image transformer pathway to explore the most discriminative facial regions. A multi-scale graph transformer network consisting of a self-attention module based on several multi-scale graphs is presented in the graph transformer pathway to adaptively extract features from the facial graph. A distillation mechanism is employed in the AGT to improve training in case of a limited number of training samples. We evaluated the proposed method on two benchmark databases: FERplus and RAF-DB. The results of comprehensive experiments on these two databases show that the proposed model can achieve comparable results to state-of-the-art methods of FER in the wild.

Automated summary

In this paper, a model called the appearance and geometry transformer (AGT) is proposed to improve the accuracy of facial expression recognition (FER) in the wild. The AGT performs feature extraction and fusion on heterogeneous data using two transformer pathways. It achieves comparable results to state-of-the-art methods on benchmark databases FERplus and RAF-DB.