Second-order training for recurrent neural networks without teacher-forcing

Neural networks with external recurrences can be successfully applied to nonlinear autoregressive moving average modeling. The process of weight adjustment is presented as a nonlinear optimization problem in the N-dimensional Euclidean space, where N is the number of adjustable weights. The least-squares criterion can be effectively minimized using a version of the conjugate gradient algorithm. Expending about the same amount of computation necessary to obtain the gradient, the required second-order information is calculated exactly. A simulation example confirms the efficacy of the training process when applied to time series prediction. Contrary to the proposed method, teacher-forced learning is shown to be ill-suited for multistep prediction