3-D Facial Feature Reconstruction and Learning Network for Facial Expression Recognition in the Wild

Facial expression recognition (FER) in the wild unavoidably suffers from the effects of face posture, illumination, and partial occlusion. In this article, we attempt to alleviate the above negative effects and improve the performance of FER in the wild based on 3-D face feature reconstruction and learning. Three-dimensional face reconstruction not only can effectively make up for the facial apparent information missing inform a 2-D face images but can also extract accurate 3-D facial geometric information in self-occlusion and extreme illumination scenarios. Therefore, we propose a novel end-to-end trainable 3-D face feature reconstruction and learning network (3-DF-RLN) is proposed for FER in the wild. In 3-DF-RLN, the 2-D implicitly frontalized face apparent data and 3-D facial landmarks are reconstructed by a 3-D face reconstruction module and input to two feature extraction pathways. The appearance pathway learns apparent features from the reconstructed 2-D face apparent data using a convolutional neural network. The geometry pathway learns the geometric features from the reconstructed 3-D facial landmarks using a graph convolutional network. Finally, FER is achieved via the fusion of the two pathways. Extensive experiments were conducted to evaluate the proposed method with three benchmark databases, including Multi-PIE, RAF-DB, and AffectNet. The results show that the proposed 3-DF-RLN model has better FER performance, both in the lab and in the wild. In addition, the face graph from the geometry pathway reveals the correlations between facial landmarks in FER.

Automated summary

In this article, a novel end-to-end trainable 3-D face feature reconstruction and learning network (3-DF-RLN) is proposed to improve the performance of facial expression recognition (FER) in the wild. Through 3-D face reconstruction, both the missing facial information and accurate facial geometric information can be effectively obtained. The proposed 3-DF-RLN model achieves FER by fusing apparent features from 2-D face images and geometric features from 3-D facial landmarks. Experimental results on benchmark databases demonstrate the superior FER performance of the proposed method, and the face graph from the geometry pathway reveals the correlations between facial landmarks in FER.