Rational approximation, harmonic analysis and neural networks

Techniques based on classical tools such as rational approximation and harmonic analysis are developed to study the computational properties of neural networks. Using such techniques, one can characterize the class of functions whose complexity is almost the same among various models of neural networks with feedforward structures. As a consequence of this characterization, for example, it is proved that any depth-(d+1) network of sigmoidal units computing the parity function of n inputs must have Ω(dn¹d-ε/) units, for any fixed ε>0. This lower bound is almost tight as one can compute the parity function with O(dn¹d/) sigmoidal units in a depth-(d+1) network. The techniques also generalize to networks whose elements can be approximated by piecewise low degree rational functions. These almost tight bounds are the first known complexity results on the size of neural networks computing Boolean functions with continuous-output elements and with depth more than two