Learning, extracting, inserting and verifying grammatical information in recurrent neural networks

Recurrent neural networks can be trained from string examples to behave like deterministic finite-state automata (DFA´s) and pushdown automata (adapts) i.e. they recognize respectively deterministic regular and context-free grammars (DCFG´s). The author discusses some of the successes and failures of this type of ´recurrent neural network´ grammatical inference engine, as well as some of the issues of effectively using a priori symbolic knowledge in training dynamic networks. The author presents a method for networks with second-order weights where inserting prior knowledge into a network becomes a straight-forward mapping (or programming) of grammatical rules into weights. A more sophisticated hybrid machine was also developed, denoted as a neural network pushdown automata (NNPDA)-a recurrent net connected to a stack memory. This NNPDA learns to operate an external stack and recognize simple DCFG´s from string examples. When hints about the grammars are given during training, the NNPDA is capable of learning more sophisticated DCFG´s.