Stochastic convergence analysis of a two-layer backpropagation algorithm for a nonlinear system identification model

The stationary points of a two-layer perceptron which attempts to identify the parameters of a specific nonlinear system are studied. The training sequence is modeled as the binary output of the nonlinear system when the input is an independent sequence of zero-mean Gaussian vectors with independent components. The training rule backpropagates the error at the input to the outer layer nonlinearity rather than the error at the output of that nonlinearity. Coupled nonlinear equations are derived for the hidden and output layer weights. These equations define a multiplicity of stationary points. One solution to these equations indicates that the hidden layer weights match those of the nonlinear system and that the outer layer weights minimize the MSE. The second layer output can be made to match the training sequence by the appropriate choice of bias. Hence, the two-layer perceptron correctly identifies the parameters of the unknown system even though the training rule does not propagate the output error