Discovering and filtering text information from Internet based on inductive learning

The paper aims at the discovering and filtering of text information from the Internet. Based on the theories of probability and statistics, an essential key word list is established by the statistics of functional words in the sample text and whether or not the text is suspicious. A probability-based inductive learning algorithm is presented to show how to set up and optimize the key words and further more classify the text by the parameter estimation of the text-class attributes. A knowledge-based network information classification and filtering scheme is studied by using the algorithm to different sets of sample text belongs to several domains. The theoretical basis and algorithm implementation are discussed in detail, the experimental results and application examples are also given in the paper