Background estimation as a labeling problem

We present a new background estimation algorithm that constructs the background of an image sequence with moving objects by copying areas from input frames. The background estimation problem is formulated as an optimal labeling problem in which the label at an output pixel is the frame number from which to copy the background color. The costs of assigning labels encourage seamless copying from regions that are stationary over a period of time in such a way that implied motion boundaries occur at intensity edges. This is accomplished without explicitly tracking the moving objects or computing optical flow. Experiments demonstrate that our algorithm is effective in difficult areas where the background is visible for only a small fraction of time, and on inputs with both moving objects that are not always in motion and moving objects with textureless areas