Restricted learning algorithm and its application to neural network training

The authors propose a new (semi)-optimization algorithm, called the restricted learning algorithm, for a nonnegative evaluating function which is 2 times continuously differentiable on a compact set Ω in R^N. The restricted learning algorithm utilizes the maximal excluding regions which are newly derived, and is shown to converge to the global ∈-optimum in Ω. A most effective application of the proposed algorithm is the training of multi-layered neural networks. In this case, one can estimate the Lipschitz´s constants for the evaluating function and its derivative very efficiently and thereby we can obtain sufficiently large excluding regions. It is confirmed through numerical examples that the proposed restricted learning algorithm provides much better performance than the conventional back propagation algorithm and its modified versions