A new error bound for the classifier chosen by early stopping

Training with early stopping is the following process. Partition the in-sample data into training and validation sets. Begin with a random classifier g₁. Use an iterative method to decrease the error rate on the training data. Record the classifier at each iteration, producing a series of snapshots g₁,…g_M . Evaluate the error rate of each snapshot over the validation data. Deliver a minimum validation error classifier, g^*, as the result of training. The purpose of the paper is to develop a good probabilistic upper bound on the error rate of g^* over out-of-sample (test) data. First, we use a validation oriented version of VC analysis (V.N. Vapnik, 1982; V.N. Vapnik and A. Chervonenkis, 1971) to develop a bound. Because of the nature of VC analysis, this initial bound is based on worst case assumptions about the rates of agreement among snapshots. In practice, though, successive snapshots are similar classifiers. We exploit this feature to develop a new bound. Then we test the bound on credit card data