Bayesian Aggregation of Binary Classifiers

Multiclass classification problems are often decomposed into multiple binary problems that are solved by individual binary classifiers whose results are integrated into a final answer. Various methods have been developed to aggregate binary classifiers, including voting heuristics, loss-based decoding, and probabilistic decoding methods, but a little work on the optimal aggregation has been done. In this paper we present a Bayesian method for optimally aggregating binary classifiers where class membership probabilities are determined by predictive probabilities. We model the class membership probability as a softmax function whose input argument is a linear combination of discrepancies between code words and probability estimates obtained by the binary classifiers. We consider a lower bound on the softmax function, which is represented as a product of logistic sigmoids, and we formulate the problem of learning aggregation weights as a variational logistic regression. Predictive probabilities computed by variational logistic regression yield the class membership probabilities. We stress two notable advantages over existing methods in the viewpoint of complexity and over fitting. Numerical experiments on several datasets confirm its useful behavior.