A Method for New Word Extraction on Chinese Large-scale Query Logs

Chinese word segmentation is a base, difficult and important problem in natural language processing. New word is a bottleneck to Chinese word segmentation. The query log of search engine contains user habits and behaviors. There are many popular words and professional terms in these logs. We present a new word extraction method, which uses the grammar rules and statistics information. The experiment was run on the Sogou query logs. Our method can achieve the precision rate of 59.9%, the recall rate of 44.1%, and the F-measure of 50.8%. Experimental results show this method has strong extensibility and good effect, and provide guarantee for improving the performance of word segment system.