Exploiting fractalness of error surfaces: New methods for neural network learning

Learning in neural networks can be formulated as global optimization of a multimodal error function over a high-dimensional space of connection weights. A general scaling model that describes the error surface as high-dimensional fraction Brownian motion (FBM), i.e., as a class of random fractals, is developed. The parameter of FBM can be extracted by spectral analysis of the error profile over a random walk in weight space. Scaling structure within the error surface has important implications for stochastic optimizations such as Boltzmann learning. Experimental data that confirm the fractalness of error surfaces for a wide range of problems and connection topologies are reviewed, and the implications of these results are discussed