Top-down pairwise potentials for piecing together multi-class segmentation puzzles

Top-down class-specific knowledge is crucial for accurate image segmentation, as low-level color and texture cues alone are insufficient to identify true object boundaries. However, existing methods such as conditional random field models (CRFs) generally impose the class-specific knowledge only at the “node” level, evaluating class membership probabilities at the (super)pixels that define the random field graph. We introduce a strategy for pairwise potential functions that capture top-down information, where we prefer to assign the same label to adjacent regions when the entropy reduction that would result from their merging is high. By measuring how the certainty of the object-level classifiers changes when considering the appearance description extracted from adjacent regions, we can “piece together” objects whose heterogenous texture would prevent both the too-local node potentials and conventional bottom-up smoothness terms from recognizing the object. We show how this idea can be used as either an affinity function for agglomerative clustering, or a pairwise potential for a CRF model. Experiments with two datasets show that the proposed entropy-guided affinity function has a clear positive impact on multi-class segmentation.