A step towards the frontier between one-hidden-layer and two-hidden-layer neural networks

This paper addresses the exact realization of functions, from the d-dimensional affine space to {0,1}, by feedforward multilayer neural networks of threshold units. A classification of the network architectures, according to their number of hidden layers, points out the difficulty of locating the frontier between dichotomies which can be realized with only one hidden layer and those which require two hidden layers. The main result is that the frontier is not directly coupled with the problem of the linear separability of Boolean functions. We give an abstract definition of a set of dichotomies that can be realized with one hidden layer. We show that this condition is sufficient but not necessary, and finally state two geometrical characterizations for dichotomies which do require two hidden layers.