On the complexity of neural networks with sigmoidal units

Novel techniques based on classical tools such as rational approximation and harmonic analysis are developed to study the computational properties of neural networks. Using such techniques, one can characterize the class of function whose complexity is almost the same among various models of neural networks with feedforward structures. As a consequence of this characterization, for example, it is proved that any depth-(d+1) network of sigmoidal units computing the parity function of n inputs must have Ω(dn^1/d-∈) units, for any fixed ∈>0. This lower bound is almost tight since one can compute the parity function with O(dn^1/d) sigmoidal units in a depth-(d+1) network. The techniques also generalize to networks whose elements can be approximated by piecewise low degree rational functions