Exploiting enhanced and robust RGB-D face representation via progressive multi-modal learning

Existing RGB-based 2D face recognition approaches are sensitive to facial variations, posture, occlusions, and illumination. Current depth-based methods have been proved to alleviate the above sensitivity by introducing geometric information but rely heavily on high-quality depth from high-cost RGB-D cameras. To this end, we propose a Progressive Multi-modal Fusion framework to exploit enhanced and robust face representation for RGB-D facial recognition based on low-cost RGB-D cameras, which also deals with in-complete RGB-D modal data. Due to the defects such as holes caused by low-cost cameras, we first design a depth enhancement module to refine the low-quality depth and correct depth inaccuracies. Then, we extract and aggregate augmented feature maps of RGB and depth modality step-by-step. Subsequently, the masked modeling scheme and iterative inter-modal feature interaction module aim to fully exploit the implicit relations among these two modalities. We perform comprehensive experiments to verify the superior performance and robustness of the proposed solution over other FR approaches on four chal-lenging benchmark databases. (c) 2022 Elsevier B.V. All rights reserved.

Automated summary

This paper proposes a progressive multi-modal fusion framework to improve and enhance RGB-D facial recognition using low-cost RGB-D cameras. The framework includes a depth enhancement module, feature extraction and aggregation steps, as well as masked modeling and inter-modal feature interaction modules. Experimental results demonstrate the superior performance and robustness of the proposed solution compared to other face recognition methods on four challenging benchmark databases.