Exploration of the correlations of attributes and features in faces

What are the attributes and features of faces that allow humans or machines to make most reliable inferences from visible to occluded regions of the face, or from shape to texture and vice versa? While both the Human Visual System and many example-based algorithms rely on correlations, these are implicit and difficult to visualize. This paper identifies and visualizes the most reliable correlations using a canonical correlation analysis (CCA) of faces in a 3D Morphable model. We investigate correlations between shape and texture, but also between shape of mouth and shape of eyes / lower and upper facial shape / overall shape and eye or mouth, and we separate intrinsic correlations from random correlations in the training set. By projecting the CCA axes on semantic attributes such as “large eyes” or “wide lips”, they can be partly translated into verbal descriptions. Using an algorithm that fills in missing information in faces, such as occluded regions, based on PCA or CCA, and a subsequent assessment of perceived similarity, we evaluate the benefit of CCA over PCA. There is no evidence of CCA being superior, which means that PCA captures correlation sufficiently and is not affected by spurious random correlations in the limited training set.