Exploiting local and global scene constraints in modeling large-scale dynamic 3D scenes from aerial video

This paper presents a framework to analyze a large amount of video data and extract high-level structural information - planar structures and motion information - in typical urban scenes, which may be used in video coding or object recognition. The method consists of two phases. In the first phase, multiple parallel-perspective (pushbroom) mosaics are generated from the video data. In the second phase, the planar structures and the moving objects are extracted from the mosaics by a segmentation-based stereo match method. The focus of this paper is the use of local and global scene constraints to improve the accuracy of high-level structural information extraction. The 3D planar patches obtained from the first step of 3D reconstruction are automatically clustered into one or more dominant planes, which are typical in indoor or outdoor city scene and are used to improve the 3D model. Then a local scene constraint is used to further refine the structure of a patch from the structures of its neighboring patches that have better structure estimations. Further, the dominant planes also provide information of road network directions, which greatly facilitates the search of moving objects on the roads. We demonstrate the effectiveness of our approach by experiments on a real video data set of a New York City scene.