Corpus-Based Statistical Screening for Content-Bearing Terms

An important problem in the indexing of natural language text is how to identify those words and phrases that reflect the content of the text. In general, automatic indexing has dealt with this problem by removing instances of a few hundred common words known as stop words, and treating the remaining words as though they were content bearing. This approach is acceptable for some applications such as statistical estimates of the similarity of queries and documents for the purpose of document retrieval. However, when the indexing terms are to be examined by a human as a means of accessing the literature, it greatly improves efficiency if most of the noncontent-bearing words and phrases can be eliminated from the indexing. Here we present three statistical techniques for identifying content-bearing phrases within a natural language database. We demonstrate the effectiveness of the methods on test data, and show how all three methods can be combined to produce a single improved method.