Coarse and fine learning in deep networks

Evolutionary systems such as Learning Classifier Systems (LCS) are able to learn reliably in irregular domains, while Artificial Neural Networks (ANNs) are very successful on problems with an appropriate gradient. This study introduces a novel method for discovering coarse structure, using a technique related to LCS, in combination with gradient descent. The structure used is a deep feature network, with a number of properties of a higher level of abstraction than existing ANNs, for example the network is constructed based on co-occurrence relationships, and maintained as a dynamic population of features. The feature creation technique can be considered a coarse or rapid initialization technique, that constructs a network before subsequent fine-tuning using gradient descent. The process is comparable with, but distinct from, layer-wise pretraining methods that construct and initialize a deep network prior to fine-tuning. The approach we introduce is a general learning technique, with assumptions of the dimensionality of input, and the described method uses convolved features. Results of classification of MNIST images show an average error rate of 0.79% without pre-processing or pretraining, comparable to the benchmark result provided by Restricted Boltzmann Machines of 0.95%, and 0.79% using dropout, however based on a convolutional topology, and as such our system is less general than RBM techniques, but more general than existing convolutional systems because it does not require the same domain assumptions and pre-defined topology. Use of a randomly initialized network provides a much poorer result (1.25%) indicating the coarse learning process plays a significant role. Classification of NORB images is examined, with results comparable to SVM approaches. Development of higher level relationships between features using this approach offers a distinct method of learning using a deep network of features, that can be used in combination with existing techniques.