Averaging and normal deviation theory for nonlinear nets

Novel techniques for solving the problem of using a mathematical model to predict neural network performance are proposed. An averaging theorem and the theory of normal deviations are applied to the stochastic equations arising from backpropagation for an arbitrary feedforward network. The averaging theorem is a method for deriving a deterministic system of equations from a stochastic system of equations, the deterministic system describing the mean behavior of a solution to the stochastic system. Applied to the specific case at hand, this allows the deviation of systems of equations which describe the mean behavior of network weights evolving through backpropagation in a noisy environment. The resulting deterministic equations can frequently be greatly simplified allowing the development of practical algorithms for prediction of the mean behavior of network weight evolution. An example of these techniques for two-layer nonlinear networks is included. The theory of normal deviations is also applied to the problem of network prediction