Title :
Simple strategies to encode tree automata in sigmoid recursive neural networks
Author :
Carrasco, Rafael C. ; Forcada, Mikel L.
Author_Institution :
Dept. de Llenguatges i Sistemes Inf., Alacanti Univ., Spain
Abstract :
Recently, a number of authors have explored the use of recursive neural nets (RNN) for the adaptive processing of trees or tree-like structures. One of the most important language-theoretical formalizations of the processing of tree-structured data is that of deterministic finite-state tree automata (DFSTA). DFSTA may easily be realized as RNN using discrete-state units, such as the threshold linear unit. A recent result by J. Sima (1997) shows that any threshold linear unit operating on binary inputs can be implemented in an analog unit using a continuous activation function and bounded real inputs. The constructive proof finds a scaling factor for the weights and reestimates the bias accordingly. We explore the application of this result to simulate DFSTA in sigmoid RNN (that is, analog RNN using monotonically growing activation functions) and also present an alternative scheme for one-hot encoding of the input that yields smaller weight values, and therefore works at a lower saturation level
Keywords :
automata theory; encoding; neural nets; recursive functions; theorem proving; tree data structures; trees (mathematics); DFSTA; adaptive processing; alternative scheme; analog RNN; analog unit; binary inputs; bounded real inputs; constructive proof; continuous activation function; deterministic finite-state tree automata; discrete-state units; language-theoretical formalizations; lower saturation level; monotonically growing activation functions; one-hot encoding; scaling factor; sigmoid RNN; sigmoid recursive neural networks; smaller weight values; threshold linear unit; tree automata encoding; tree-like structures; tree-structured data; Analog computers; Automata; Computational modeling; Computer Society; Encoding; Intelligent networks; Neural networks; Neurons; Recurrent neural networks; Tree graphs;
Journal_Title :
Knowledge and Data Engineering, IEEE Transactions on