Joint learning of foreground region labeling and depth ordering

This paper considers a joint learning algorithm of foreground region labeling and depth ordering for 3D scene understanding. Given an object-level segmentation, the proposed algorithm classifies each region as either foreground or background while simultaneously infers the relative depth orders between every adjacent region pairs. For this, we consider a graph where regions are considered as nodes while boundaries between adjacent regions as edges, and the problem is formulated as jointly assigning binary labels to every nodes and edges via maximizing a unified linear discriminant function, under the constraints that make the resulting depth order to be always physically plausible. Instead of inferring region and edge labels separately, we infer them jointly by grouping them as a single variable referred to as triplet. Then, the problem is reformulated as multi-class triplet prediction to penalize the inconsistent labeling of regions and edges in a soft manner. As the discriminant function is linear, the parameters can be learned with structured support vector machine(S-SVM), and efficient inference using linear programming relaxation is possible. Experimental results show that the proposed joint inference algorithm improves both foreground region labeling and depth ordering performances.