Powell´s dogleg trust-region steps with the quasi-Newton augmented Hessian for neural nonlinear least-squares learning

This paper highlights Powell´s dogleg trust-region algorithms with self-scaling quasi-Newton Hessian augmentation for neural-network (NN) nonlinear least squares problems. The dogleg algorithms approximate a restricted Levenberg-Marquardt step within the trust region of the local quadratic model in a piecewise-linear fashion. Furthermore, the second-derivative term of the Hessian is approximated by quasi-Newton iteration to obtain augmented Gauss-Newton model Hessian, which may be useful for highly nonlinear residuals when starting with a poor initial point (i.e., randomly initialized weight parameters). By small-scale examples, we illustrate how those devices come into play as a promising NN learning algorithm