On minimal net size, convergence speed and generalization ability of backpropagation trained heterogeneous neural net

A three-layer heterogeneous network consisting of linear output layer units and sigmoid hidden layer units is considered. Compared with a conventional homogeneous network of all sigmoid units, this network can be implemented much more economically, and for analog mapping this network is more suitable. The minimal hidden size needed for convergence in a three-layer heterogeneous network for a given task is derived and a performance comparison between the two types of networks in terms of convergence behavior and generalization ability is also presented. The comparison shows that the heterogeneous network has the unique feature of very fast early stage learning and converges much faster for a suitable range of the slope of the learning activation function. Both have comparable generalization abilities