Background recovery from multiple images

In this paper we propose an algorithm to extract the background by removing unwanted objects from multiple images of a scene with varying illumination conditions captured by a stationary camera. The variations in illumination from scene to scene are due to the possible presence of different illumination sources and different foreground objects causing different shadows and reflections in each scene. While this causes the background to be non-stationary when considering pixel intensities, the algorithm exploits the fact that the background is static while the foreground is non-stationary in a given feature space. The extracted foreground regions are treated as holes and are filled from one of the available images where the background at the corresponding position is un-occluded. The background pixels for filling the holes are selected based on a cost function that attempts to maximize the naturalness and perceived quality of the reconstructed background.