Search-Engine-Oriented Theme Crawler Design

A theme crawler is the most important part of a vertical search engine. To recall web pages efficiently and accurately, the design work of theme crawler was studied in this paper. Seed link and similarity measurement are two key techniques for a theme crawler, which are explained in detail in this paper. And the relevant program codes and algorithm were provided to explain there two techniques clearly. The process of a theme crawler begins from fetching seed links, host search engine, interface of search engine and fetch link were illustrated in the paper. To improve the efficiency of crawler, a model of page evaluation was added to the crawler module.