Optimal use of regularization and cross-validation in neural network modeling

This paper proposes a new framework for adapting regularization parameters in order to minimize validation error during the training of feedforward neural networks. A second derivative of validation error based regularization algorithm (SDVR) is derived using the Gauss-Newton approximation to the Hessian. The basic algorithm, which uses incremental updating, allows the regularization parameter α to be recalculated in each training epoch. Two variations of the algorithm, called convergent updating and conditional updating, enable α to be updated over a variable interval according to the specified control criteria. Simulations on a noise-corrupted parabolic function with two-inputs and a single output are investigated. The results demonstrate that the SDVR framework is very promising for adaptive regularization and can be cost effectively applied to a variety of different problems