A hybrid method to segment words

Word segmentation is the foundations of machine translation, text classification and information searching. A method is proposed which combines word segmentation based on dictionary with reverse maximum matching and word segmentation based on statistic with suffix array. The input texts are segmented using the reserve maximum matching method based on dictionary, and a two-way suffix arrays are constructed, longest common prefix are computed, candidate words are filtered out by setting the threshold, the candidate words are filtered using mutual information in order to the true words. The texts that are ambiguity are filtered using information entropy. It is showed that the accuracy of word segmentation may achieve above 97% in the experiment.