Joint space-time image sequence segmentation: object tunnels and occlusion volumes

Spatial segmentation of image sequences is usually computed based on motion between two frames. Some recent approaches extend this to joint segmentation in space-time; the resulting 3D segmentation (in x-y-t space) can be interpreted as a volume "carved out" by a moving object in the image sequence domain, or the so-called "object tunnel". We extend this concept to explicit modeling of occlusion events in the x-y-t space. In addition to the modeling of object evolution, we also model occluded and newly-exposed areas in the background and in the object by means of an "occlusion volume", a new space-time concept. We propose a variational formulation of the problem that we solve using the multiphase level set method. We show experimental results for synthetic and natural image sequences.