Improving generalisation with Ockham´s networks: minimum description length networks

There exists a substantial problem in obtaining good generalisation performance in the application of artificial neural network technology where training data is limited. A number of current techniques aiming to improve generalisation are introduced from the perspective of the minimum description length (MDL) principle. These are quadratic weight decay, soft weight-sharing and the technique introduced by the authors, Ockham´s networks. In addition to presenting the major developments of Ockham´s networks, a summary of a case study comparing these techniques is presented. It is found that Ockham´s networks provide an improvement in generalisation performance as good as any other technique tested in addition to using the smallest number of weights