Out-of-core backpropagation

Backpropagation learning can execute at supercomputer speed from training data sets of unprecedented size when supercomputer main memory is backed with newly available parallel arrays of commodity disk drives. An efficient implementation of backpropagation learning was modified and extended to iterate through training data sets stored on a parallel-disk backing store. The algorithm is standard, including generating and adding noise to training inputs and the usual momentum term. With data sets up to the 10-GB capacity of the backing store, this backpropagation executes on the 16384-processor Connection Machine (CM) and parallel disks at 9.3 million connections per second (MCPS). A rate of 31 MCPS on a 65536-processor CM is predicted. This backpropagation reliably and automatically trained a feedforward, two-hidden-layer artificial neural network classifier with 33824 weights using 67584 input-output training pairs. This practical, forward-and-backward training computation through one pass of the data executed in 4.1 min. Moving the training from data parallel disks to main memory took 6.5% of the execution time. On a 65536-processor CM, a time of 74 s, with 22% spent on this data movement, is projected