Developing an efficient cross validation strategy to determine classifier performance (CVCP)

We develop an efficient cross validation strategy to determine classifier performance (CVCP). Previous techniques for LOO CV (leave one out cross validation) required the evaluation of up to M models, with M the sample size, making it prohibitively expensive for many practical applications. We present a technique, namely CVCP, requiring only O(Mlog2(M)) operations and show that the performance of the classifier can be evaluated accurately with the new technique. To combat overfitting of neural classifiers CV is often used to determine the performance on an independent test set. An extreme form of validation is LOO CV, in which each sample forms an independent test set. In this way very efficient use is made of the data set. Unfortunately this implies that we need to build M models. Due to the slow convergence of neural network architectures, it may not be practical to use LOO CV in many applications. We propose an efficient algorithm, CVCP, to determine CV performance. The resulting error rate compares favourably with the Bayes error rate. CVCP can thus be used to determine classifier performance without the associated computational complexity of LOO CV