Spatio-temporal segmentation with Mumford-Shah functional

Image segmentation is intended to group perceptually similar pixels into 2D regions, and the corresponding border is gained at the same time. Video segmentation generalizes this concept to the grouping of pixels into spatio-temporal regions that exhibit coherence in both appearance and motion, but this generalization pose the complexity of spatio-temporal grouping, and in order to overcome this complexity, the existing video segmentation methods have extended the image segmentation methods to 3D domain. In these volumetric approaches, it is not known a priori, which regions to track, what frames contain those regions, or the time-direction for tracking (forward or backward). In this paper we present an efficient and scalable method for spatio-temporal segmentation obtained by minimizing a 2D+time extension of the simplified Mumford-Shah functional. The 2D+time extension permits to write the Mumford-Shah functional as an a multiscale energy, which is minimized on a hierarchy of video domain. The construction of this hierarchy based on the 2D-shapes of video images, and Scale-invariant feature transform (SIFT).