Optimal Size of a Feedforward Neural Network: How Much does it Matter?

In this paper, we attempt to answer the following question with systematic computer simulations: for the same validation error rate, does the size of a feedforward neural network matter? This is related to the so-called Occam´s Razor, that is, with all things being equal, the simplest solution is likely to work the best. Our simulation results indicate that for the same validation error rate, smaller networks do not tend to work better than larger networks, that is, Occam´s Razor does not seem to apply to feedforward neural networks. In fact, our results show no trend between network size and performance for a given validation error