Study on Distributed Representation of Words with Sparse Neural Network Language Model

These days a neural network is paid attention to again since it is improved as deep learning. Deep learning achieves good data representation according to a data distribution and get over state of the art classifiers in computer vision and speech recognition. The representation captures abstracts of many data and is used as general features to solve many types of classification problems. A neural network is applied to natural language processing, too. In natural language processing neural networks achieve the best distributed representation of words and many researchers pay attention to the neural network language model. The distributed representation allocates words in a continuous feature space and there are semantically or syntactically similar words near area in the space. Hence, the distributed representation contributes to a solution of analogical reasoning tasks. In this paper a sparse neural network language model (SNNLM) is used, which achieves sparse active neurons in the hidden layer and a distributed representation of words is obtained. In evaluational experiments SNNLM selects words that do not occur in the same sentence at all as related words and it is confirmed that the word selection is appropriate manually.