Mode committee: a novel ensemble method by clustering and local learning

Ensemble methods have proved effective to achieve higher accuracy. Some simple ensemble methods, such as Bagging, work well with unstable base algorithms, but fail with stable ones. The reason is that such methods achieve higher accuracy by reducing only the variance of the base algorithms. It does not touch the bias. Here, we propose a novel ensemble method, mode committee, intended to work for both stable and unstable base algorithms. It first derive a new algorithm, called mode competitor, from given base algorithm, with the help of k-modes clustering method and the local learning strategy. Randomness is injected into each mode competitor by the process of random seeding. The aim of deriving mode competitor is to reduce the bias with the possible increasing variance. Then, multiple mode competitors form a committee and vote on the decision of new example, with the aim to reduce the variance of mode competitors. Such an arithmetic framework has been materialized by two base algorithms, the unstable C4.5 and the stable naive Bayes. Extensive empirical results demonstrate this method´s superiority, and further analysis by bias-variance decomposition reveals that it is due to the low-bias of mode competitors.