An improved Global Weight Function of Terms based on Pearson´s Chi-square statistics

Since term frequency, the most popular discriminator used in term weighting of Natural language processing (NLP), is not the only one which is necessary to be considered when calculating the term weight and make it suitable to indicate term importance, we are motivated to investigate other statistical characteristics of terms and found an important discriminator: term distribution. It is found in this this paper that a term close to hypo-dispersion distribution usually contains much contextual information and should be given higher weight than the one close to intensive distribution. Based on this hypothesis, a Pearson´s Chi-square Theory based Term Global Weight Function is put forward in this paper. In addition, a text classifier system is developed based on LSA (Latent Semantic Analysis) model and its precision and recall results are used for evaluation, which approve the reliability and efficiency of the algorithm On conclusion, term distribution should be considered into term weighting as a new discriminator and the algorithms in this paper is recommended.