Finding optimal neural network basis function subsets using the Schmidt procedure

In designing feedforward neural networks, one often trains a large network and then prunes less useful hidden units. In this paper, two non-heuristic pruning algorithms are derived from the Schmidt procedure. In both, orthonormal systems of basis functions are found, ordered, pruned, and mapped back to the original network. In the first algorithm, the orthonormal basis functions are found and ordered one at a time. In optimal pruning, the best subset of orthonormal basis functions is found for each size network. Linear dependency of basis functions is considered and computational cost is analyzed. Simulation results are given.