A comprehensive study of the backpropagation algorithm and modifications

Many connectionist/neural network learning systems use some derivative of the popular backpropagation (BP) algorithm. BP learning, however, is too slow for many applications. In addition, it scales poorly as tasks become larger and more complex. As a result, researchers in the field have come up with variations and modifications to the original BP learning technique that address the aforementioned issues. This research was conducted to collect a representative sample of BP modifications and compare them against one another. The benchmarks utilized are certain "toy-problems" that have been extensively used in the literature. A software package that allows one to experiment with a multitude of BP variations was developed to achieve the desired goal. The modifications are evaluated and cross examined for each task tested. The package provides the means for parameter optimization and allows a user to build hybrid algorithms based on the different functionalities and features of the various modifications.