High-performance neural network training on a computational cluster

In this paper, a high performance neural network for image segmentation is described which utilizes a Myrinet-based PC cluster to implement back propagation training on large data sets. Back propagation training of the very large neural network described here requires substantial computing resources that impact the time required for training. Work here shows that exploitation of modern microprocessor architectures as well as the distribution of processing over a high performance cluster can significantly impact training times required and effectively enlarge the envelop within which large neural networks remain computationally feasible. This work proposes a personal computer cluster solution that balances the distribution of computations while minimizing communication among cluster nodes. Results show that for a PC cluster with n nodes, the training of neural networks running on the cluster has a speedup of near (n-2) times when compared to training on a single node.