Tracking with context as a semi-supervised learning and labeling problem

It is suggested how a Markov random field can be used for object tracking with context information. The tracking is formulated as a two layer process. In the first phase, the image is represented by a set of feature points which are tracked by a standard tracker. In the second phase, the proposed semi-supervised learning and labeling algorithm is used to label the points to three classes - object, background and companion. The object state (pose) is defined by the set of points labeled as the object. The companion represents the object context and contains non-object points with a motion similar to the motion of the object. As initialization, labels of the object points only are provided by a user in the very first frame. The appearance and motion models of the three classes and the labels of the remaining points in the whole video sequence are estimated in a GrabCut fashion. We show that the use of the companion class together with a 3D (space-time) Markov random field helps to identify object points behind full occlusions or under strong appearance changes.