Joint Holistic and Masked Face Recognition

With the widespread use of face masks due to the COVID-19 pandemic, accurate masked face recognition has become more crucial than ever. While several studies have investigated masked face recognition using convolutional neural networks (CNNs), there is a paucity of research exploring the use of plain Vision Transformers (ViTs) for this task. Unlike ViT models used in image classification, object detection, and semantic segmentation, the model trained by modern face recognition losses struggles to converge when trained from scratch. To this end, this paper initializes the model parameters via a proxy task of patch reconstruction and observes that the ViT backbone exhibits improved training stability with satisfactory performance for face recognition. Beyond the training stability, two strategies based on prompts are proposed to integrate holistic and masked face recognition in a single framework, namely FaceT. Along with popular holistic face recognition benchmarks, several open-sourced masked face recognition benchmarks are collected for evaluation. Our extensive experiments demonstrate that the proposed FaceT performs on par or better than state-of-the-art CNNs on both holistic and masked face recognition benchmarks. Codes will be made available at https://github.com/zyainfal/Joint-Holistic-and-Masked-Face-Recognition.

Automated summary

With the increase in the use of face masks due to COVID-19, accurate recognition of masked faces has become extremely important. This paper explores the use of plain Vision Transformers (ViTs) for masked face recognition, which has been less researched compared to convolutional neural networks (CNNs). The paper proposes a model initialization method and two prompt-based strategies to integrate holistic and masked face recognition. Experimental results show that the proposed FaceT performs as well as or better than state-of-the-art CNNs on both holistic and masked face recognition benchmarks.