Intensity enhancement via GAN for multimodal face expression recognition

Face expression recognition (FER) on low expression intensity is not well studied in the literature. This paper investigates this problem and presents a novel Generative Adversarial Network (GAN) based multimodal approach to it. The method models the tasks of intensity enhancement and expression recognition jointly, ensuring that the synthesize faces not only present expression of high intensity, but also truly promote the performance of FER. The proposed model is flexible enough that faces can be expressed in various formats, such as RGB image, depth maps, 3D point-clouds, etc., so that complementarity of texture and geometry clues can be further exploited. Extensive experiments are conducted on the BU-3DFE, BU-4DFE, Oulu-CASIA and CK+ datasets. State-of-the-art FER performance is achieved for not only the circumstance of low expression intensities, but also the general FER scenarios, clearly validating the effectiveness of the proposed method. (c) 2021 Published by Elsevier B.V.

Automated summary

This paper proposes a novel multimodal approach based on GAN that jointly models intensity enhancement and expression recognition, resulting in improved FER performance. Experimental results on multiple datasets validate the effectiveness of the method in both low expression intensity and general FER scenarios.