Neural network architectures and learning

Various learning methods of neural networks including supervised and unsupervised methods are presented and illustrated with examples. General learning rule as a function of the incoming signals is discussed. Other learning rules such as Hebbian learning, perceptron learning, LMS (least mean square) learning, delta learning, WTA (winner take all) learning, and PCA (principal component analysis) are presented as a derivation of the general learning rule. Architecture specific learning algorithms for cascade correlation networks, Sarajedini and Hecht-Nielsen networks, functional link networks, polynomial networks, counterpropagation networks, RBF (radial basis function) networks are described. Dedicated learning algorithms for on chip neural network training are also evaluated. The tutorial focuses on various practical methods such as Quickprop, RPROP, Back Percolation, Delta-bar-Delta and others. Main reasons of convergence difficulties such as local minima or flat spot problems are analyzed. More advance gradient-based methods including pseudo inversion learning, conjugate gradient, Newton and LM (Levenberg-Marquardt) algorithm are illustrated with examples.