Chinese Word Segmentation as POC-NLW Tagging

In Chinese word segmentation, disambiguation and unknown words identification are the two key issues still remaining. In order to deal with these problems in a uniform way, a language tagging template, named POC-NLW, is presented in this paper to explore the word creation mechanisms of Chinese language on character-level. Based on this template, a hidden Markov model based tagger is constructed to implement word segmentation as character tagging. In this method, the basic word segmentation, disambiguation, and the unknown words identification are integrated fundamentally and accomplished in one unified process. Experimental results on the SIGHAN Bakeoff2005 corpus show that the method can achieve high accuracy on word segmentation, especially on unknown words identification, with appreciable processing efficiency. This method is characterized by the good interoperability and expansionary over different kinds of words, thus it is applicable for practical Chinese information processing applications