RGB-D-Based Face Reconstruction and Recognition

Most RGB-D-based research focuses on scene reconstruction, gesture analysis, and simultaneous localization and mapping, but only a few study its impacts on face recognition. A common yet challenging scenario considered in face recognition takes a single 2D face of frontal pose as the gallery and other poses as the probe set. We consider a similar scenario but with an RGB-D image pair taken at frontal pose for each subject in the gallery, only 2D images with a large scope of pose variations in the probe set, and study the advantage of the additional depth map on top of the regular RGB image. To tackle the cases with depth map corrupted by quantization noise, which are often encountered when the face is not close enough to the RGB-D camera, we propose a resurfacing approach as a preprocessing phase. We formulate the 3D face reconstruction using the RGB-D image as a constrained optimization and compare the results with different reconstruction settings. The reconstructed 3D face allows the generation of 2D face with specific poses, which can be matched against the probes. To deal with occlusion and expression variations, an automatic landmark detection algorithm is exploited to identify the parts on a given probe that are good for recognition. Experiments on benchmark databases show that the additional depth map substantially improves the cross-pose recognition performance, and the landmark-based component selection also improves the recognition under occlusion and expression variation. The performance comparison with other contemporary approaches also shows the effectiveness of the proposed approach.