Learning Non-linear Calibration for Score Fusion with Applications to Image and Video Classification

Image and video classification is a challenging task, particularly for complex real-world data. Recent work indicates that using multiple features can improve classification significantly, and that score fusion is effective. In this work, we propose a robust score fusion approach which learns non-linear score calibrations for multiple base classifier scores. Through calibration, original base classifiers scores are adjusted to reflect their true intrinsic accuracy and confidence, relative to the other base classifiers, in such a way that calibrated scores can be simply added to yield accurate fusion results. Our approach provides a unified approach to jointly solve score normalization and fusion classifier learning. The learning problem is solved within a max-margin framework to globally optimize performance metric on the training set. Experiments demonstrate the strength and robustness of the proposed method.