Improving the reliability of decision tree and naive Bayes learners

The C4.5 decision tree and naive Bayes learners are known to produce unreliable probability forecasts. We have used simple binning (Zadrozny and Elkan, 2001) and Laplace transform (Cestnik, 2001) techniques to improve the reliability of these learners and compare their effectiveness with that of the newly developed Venn probability machine (VPM) meta-learner (Vovk et al., 2003). We assess improvements in reliability using loss functions, receiver operator characteristic (ROC) curves and empirical reliability curves (ERC). The VPM outperforms the simple techniques to improve reliability, although at the cost of increased computational intensity and slight increase in error rate. These trade-offs are discussed.