Robust 3D face modeling and tracking from RGB-D images

We address the issue of 3D face modeling and tracking from RGB-D images. Existing methods usually fit a deformable model to an RGB-D image using iterative closest point algorithm. Due to the noise and occlusion of the depth image, these methods are not robust enough. To solve this issue, we propose a method for robust 3D face modeling and tracking. For an input RGB-D face image, our method first estimates the initial head pose of a person using random forests. Then, a generic bilinear face model is fitted to the RGB-D image using iterative closest point algorithm. To improve the accuracy and robustness of face modeling, an optimal weight for each face vertex is integrated into the fitting procedure. The distances between facial landmarks are also used to better estimate facial expressions. Finally, the head pose, the identity, and expression parameters of the bilinear face model are jointly optimized. Experiments show that our method can generate accurate 3D face models from an RGB-D image or image sequence. The method can also robustly track the face even if the person is with large head rotations and various facial expressions.

Automated summary

This paper addresses the problem of robust 3D face modeling and tracking from RGB-D images. The proposed method estimates the initial head pose using random forests and fits a generic bilinear face model to the RGB-D image using the iterative closest point algorithm. It improves the accuracy and robustness by integrating optimal vertex weights and utilizing the distances between facial landmarks. The experimental results demonstrate that the method can generate accurate 3D face models and track the face robustly under large head rotations and various facial expressions.