Analysis and improvement of multiple optimal learning factors for feed-forward networks

The effects of transforming the net function vector in the multilayer perceptron are analyzed. The use of optimal diagonal transformation matrices on the net function vector is proved to be equivalent to training the network using multiple optimal learning factors (MOLF). A method for linearly compressing large ill-conditioned MOLF Hessian matrices into smaller well-conditioned ones is developed. This compression approach is shown to be equivalent to using several hidden units per learning factor. The technique is extended to large networks. In simulations, the proposed algorithm performs almost as well as the Levenberg Marquardt algorithm with the computational complexity of a first order training algorithm.