On the methodology for comparing learning algorithms: a case study

We explore several issues relevant to the benchmarking and comparison of machine learning algorithms. We illustrate those issues with a case study using the decision tree induction algorithms C4.5 (J. Quinlan, 1993) and multiscale classification (MSC) (A.P. Bradley and B.C. Lovell, 1994), multilayer perceptrons (MLP) and multivariable regression (MVR). Then for a “real world” problem we compare estimates of the true error rates for each classifier, first on a single train and test partition, and then using cross validated subsampling techniques. The relevance of the χ² test is then discussed in relation to comparing the classifier accuracies. The paper concludes by evaluating the performance of these four fundamentally different approaches to the solution of this regression problem