Estimating all frequency lighting using a color/depth image

This paper presents a novel approach to estimating the lighting using a pair of one color and one depth image. To effectively model all frequency lighting, we introduce a hybrid representation for lighting; the combination of spherical harmonic basis functions and point lights. Upon the existing framework of spherical harmonics based diffuse reflection, we divide the color image into diffuse and non-diffuse reflections. Then, we use the diffuse reflection for estimating the low frequency lighting. For high frequency lighting, we obtain the specular reflections by analyzing the non-diffuse reflection. Knowing specular reflections and scene geometry, we are capable of computing the direction of point lights, inverting the reflected ray with respect to the surface normal. Then, we optimize the intensity of point lights by analysis by synthesis paradigm. By superimposing the low and high frequency lighting, we recover the lighting present in the scene. While existing methods use the low frequency lighting to infer the high frequency lighting, we propose to use the nondiffuse reflection for directly estimating the high frequency lighting. In this way, we make good use of the non-diffuse reflections in scene analysis and understanding. Experimental results show that the proposed approach is an effective solution for the lighting estimation of real world environment.