AOSA-LogistBoost: Adaptive One-Vs-All LogistBoost for multi-class classification problems

We present a general framework for constructing adaptive LogistBoost Multi-Class Classification algorithms. In contrast to the original LogistBoost algorithm, which constructs J (the number of classes) scalar regression trees per iteration, we construct a single vector regression tree per iteration. Based on the analysis of the loss function, the concept of node mode and the AOSA-LogistBoost (Adaptive One-vS-All LogistBoost) algorithm are proposed. We perform numerical experiments on ten datasets to test the performance of the new algorithm. Compared to the original LogistBoost algorithm, the AOSA-LogistBoost has faster convergence rate and lower test error rate. Furthermore, we introduce the inverted index to quickly re-sort the data samples after node split with linear computational complexity. Methods like Bagging, Random Features and Random Forest, are shown to be able to improve the performance of the decision tree. We also test the performance of combining Bagging and Random Features with our algorithm.