3-D wafer scale architectures for neural network computing

We introduce a class of massively parallel computer architectures which can be configured to efficiently handle a variety of neural network models. The underlying technology is three-dimensional wafer scale integration (3-D WSI), which provides an ideal medium to construct powerful, compact, and low power hardware tailored for neural network processing. A second generation prototype computer consisting of a 128×128 array of processors formed by stacking 8 CMOS wafers is nearing completion. The performance of this prototype is compared with enhanced architectures configured with special wafer types to accelerate neural network operations. The design of these specialized resources emphasizes the synergy between neural processing functions and the 3-D WSI architecture and packaging. Detailed microcode emulations are used to assess the impact of different algorithm/architecture modifications. Neural networks for cooperative vision integration and multilayer backpropagation are mapped onto various 3-D wafer stacks. Estimated performance for the vision integration network is 2.4 billion connections per second. For the backprop network training algorithm, the performance ranges from 1.1 billion connection updates per second (GCUPS) for a near-term 128×128 prototype up to 53.4 GCUPS for a future 512×512 machine with more extensive neural processing hardware enhancements