Smoothing backpropagation cost function by delta constraining

Convergence problems in the case of the generalized delta rule are discussed. A modification to the nonlinearity of processing elements is proposed which is shown to smooth the cost function to minimized during the learning phase. A variation to the generalized delta rule learning procedure, required by the introduced modification, is discussed. Extensive tests have been performed on several examples proposed in the technical literature. The tests show the effectiveness of the proposed procedure in improving the convergence properties of the backpropagation algorithm. In particular, it was shown that the proposed modification virtually eliminates nonconvergence problems if a moderate η value is used