Sandwich Cut: An Algorithm for Temporally-Coherent Video Bilayer Segmentation

In this paper, we introduce the Sandwich Cut: an algorithm for temporally-coherent bilayer segmentation in monocular video sequences. Building upon the observation that temporally-coherent segmentation relys on the frame with its temporal neighbors, the key idea is looking ahead to a future frame for the current segmentation by constructing a three-frame graph like a sandwich. The sandwich, three-frame graph, combines the cues of previous segmentation, color, spatial and temporal color differences, where interlayer weights are estimated by temporal gradients. In each step of our sequential system the final result of the current frame and the temporary result of the future one are obtained, and the latter is used to estimate an adaptive coherence factor for the next sandwich cut. The presentation of the system is complemented with the quantitative evaluations and comparisons with the latest techniques on a public database. Moreover, a method for evaluating the temporal coherence of the results is introduced and tested. Compared with the classic optical flow based method, our system performances better with higher efficiency.