A hierarchical, context-dependent neural network architecture for improved phone recognition

In this paper we combine three simple refinements proposed recently to improve HMM/ANN hybrid models. The first refinement is to apply a hierarchy of two nets, where the second net models the contextual relations of the state posteriors produced by the first network. The second idea is to train the network on context-dependent units (HMM states) instead of context-independent phones or phone states. As the latter refinement results in a lot of output neurons, combining the two methods directly would be problematic. Hence the third trick is to shrink the output layer of the first net using the bottleneck technique before applying the second net on top of it. The phone recognition results obtained on the TIMIT database demonstrate that both the context-dependent and the 2-stage modeling methods can bring about marked improvements. Using them in combination, however, results in a further significant gain in accuracy. With the bottleneck technique a further improvement can be obtained, especially when the number of context-dependent units is large.