Evaluation of robustness and performance of Early Stopping Rules with Multi Layer Perceptrons

In this paper, we evaluate different early stopping rules (ESR) and their combinations for stopping the training of multi layer perceptrons (MLP) using the stochastic gradient descent, also known as online error backpropagation, before reaching a predefined maximum number of epochs. We focused our evaluation to classification tasks, as most of the works use MLP for classification instead of regression. Early stopping is important for two reasons. On one hand it prevents overfitting and on the other hand it can dramatically reduce the training time. Today, there exists an increasing amount of applications involving unsupervised and automatic training like i.e. in ensemble learning, where automatic stopping rules are necessary for keeping training time low. Current literature is not so specific about endorsing which rule to use, when to use it or what its robustness is. Therefore this issue is revisited in this paper. We tested on PROBEN1, a collection of UCI databases and the MNIST.