Adopting Wildlife Experiments for Web Evolution Estimations: The Role of an AI Web Page Classifier

This paper proposes a statistical approach for estimating the evolution of Web pages in directories. The proposal is based on the capture-recapture method used in wildlife biological studies in an animal, bird or fish populations, and it is modified according to the necessary assumptions and amendments for applying the experiments in a search engine directory. During these experiments, Web pages are considered as animals and the specific types of Web pages as particular species of animals whose abundance, birth, death and survival rates are estimated. The population is open, meaning that new Web pages are submitted to the search engine directory, while others are removed from the directory indexes, resembling to emigration/immigration processes in nature. The role of the biologist who recognizes the species under study and records their history is assigned to a Web page classifier, which is trained under the open directory´s (DMOZ project) taxonomy. The classifier is a three layer probabilistic neural network capable of identifying and categorizing Web pages, on the basis of information filtering. A virtual experiment is simulated based on the classifier performance over real Web pages, while the results are quite promising