Gabor-based patch covariance matrix for face sketch synthesis

In this paper, we propose a novel face sketch/photo synthesis method by utilizing Gabor-based Patch Covariance Matrix (GPCM) as face descriptor, a.k.a. symmetric positive definite matrix, which lie on a Riemannian manifold. In particular, both pixel locations and Gabor coefficients of one patch are employed to form the covariance matrix. In this way, the sketch/photo can be then transformed from the pixel space to the Riemannian manifold space. With the aid of the recently introduced Stein kernel theory, we advance to perform Regularized Least Square Representation (RLSR) in Stein space. Based on the assumption that the Stein divergence manifold of photo/sketch patch and the sketch/photo share the same topology, a new sketch/photo patch of the same position can be synthesized by keeping the weights and replacing the photo/sketch training image patches with the corresponding sketch/photo ones. Experimental results demonstrate the superiority of the proposed method.