Depth-assisted nonlocal means hole filling for novel view synthesis

In novel view synthesis using the depth image based rendering, there exist some unknown pixel intensities (holes) due to the unexposed area. In this paper, we propose a depth-assisted nonlocal means algorithm to fill the holes using the information in the current frame and other frames in the synthesized video. The nonlocal means has been successfully applied to the video super-resolution applications. In hole filling, the challenge is the irregular interpolation. However, there can be a relatively reliable depth map, such that we are able combine the irregular intensity information and the reliable depth information, and make use of the formulation of nonlocal means to fill the holes. Experimental results show that the proposed algorithm outperforms the conventional spatial and temporal approaches in terms of visual quality.