Multiclass pixel labeling with non-local matching constraints

A popular approach to pixel labeling problems, such as multiclass image segmentation, is to construct a pairwise conditional Markov random field (CRF) over image pixels where the pairwise term encodes a preference for smoothness within local 4-connected or 8-connected pixel neighborhoods. Recently, researchers have considered higher-order models that encode soft non-local constraints (e.g., label consistency, connectedness, or co-occurrence statistics). These new models and the associated energy minimization algorithms have significantly pushed the state-of-the-art for pixel labeling problems. In this paper, we consider a new non-local constraint that penalizes inconsistent pixel labels between disjoint image regions having similar appearance. We encode this constraint as a truncated higher-order matching potential function between pairs of image regions in a conditional Markov random field model and show how to perform efficient approximate MAP inference in the model. We experimentally demonstrate quantitative and qualitative improvements over a strong baseline pairwise conditional Markov random field model on two challenging multiclass pixel labeling datasets.