Dimensionality Reduction using a Mixed Norm Penalty Function

The dimensionality of a problem that is addressed by neural networks is related to the number of hidden neuron in the network. Pruning neural networks to reduce the number of hidden neurons reduces the dimensionality of the system, produces a more efficient computation and yields a network with better ability to generalize beyond the training data. This work introduces a novel penalty function that is shown to reduce the number of active neurons. The performance of this function is superior to other known penalty functions. To best implement this function, we use bi-level optimization, which enables us to reduce dimensionality while maintaining good classification performance