Chinese Word Segmentation and Out-of-Vocabulary Words Detection Using Suffix Array

At present Chinese word segmentation roughly consists of two mature methods: dictionary-based method and statistical method. Both methods have their own relative merits. To achieve a better performance, this paper proposes an algorithm using amended dictionary integrating with suffix array to detect OOV words. The essential idea is to accurately and efficiently extract medium and high frequency lexical items by using suffix array, and simultaneously extract other items using the amended dictionary. We mend the storage structure of the dictionary to improve the speed of segmenting. The experiment has proved that this method can improve the integrity and the accuracy effectively.