Speeding-up backpropagation-a comparison of orthogonal techniques

In recent years much effort has been spent trying to develop more efficient variations of the backpropagation learning algorithm. This has led to a combinatorial explosion of learning methods of which no detailed evaluation exists. We have analyzed the most important algorithms and extracted their minimal building blocks. By arranging these building blocks in different forms, and testing the resulting algorithms, we obtained new combinations which were benchmarked in a commercial workstation. Our results show which factors are responsible for the increased speed-up of the tested algorithms. These results could lead to better learning methods for neural networks.