Multiple foreground recognition and cosegmentation: An object-oriented CRF model with robust higher-order potentials

Localizing, recognizing, and segmenting multiple foreground objects jointly from a general user´s photo stream that records a specific event is an important task with many useful applications. As argued in recent Multiple Foreground Cosegmentation (MFC) work by Kim and Xing, this task is very challenging in that it contrasts substantially from the classical cosegmentation problem, and aims to parse a set of realistic event photos but each containing irregularly occurring multiple foregrounds with high appearance and scene configuration variations. Inspired by the impressive advance in scene understanding and object recognition, this paper casts the multiple foreground recognition and cosegmentation (MFRC) problem within a conditional random fields (CRFs) framework in a principled manner. We capitalize centrally on the key objective that MFRC is to segment out and annotate foreground objects or “things” rather than “stuff”. To this end, we exploit a few complementary objectness cues (e.g. contours, object detectors and layout) and propose novel and efficient methods to capture object-level information. Integrating object potentials as soft constraints (e.g. robust higher-order potentials defined over detected object regions) with low-level unary and pairwise terms holistically, we solve the MFRC task with a probabilistic CRF model. The inference for such a CRF model is performed efficiently with graph cut based move making algorithms. With a minimal amount of user annotations on just a few example photos, the proposed approach produces spatially coherent, boundary-aligned segmentation results with correct and consistent object labeling. Experiments on the FlickrMFC dataset justify that our method achieves state-of-the-art performance.