The Improved Pagerank in Web Crawler

Pagerank is an algorithm for rating web pages. It introduces the relationship of citation in academic papers to evaluate the web page´s authority. It gives the same weight to all edges and ignores the relevancy of web pages to the topic, resulting in a problem of topic-drift. On the analysis of several pagerank algorithms, an improved pagerank based upon thematic segments is proposed. In this algorithm, a web page is divided into several blocks by Html document´s structure and the most weight is given to linkages in the block that is most relevant to given topic. Moreover, the visited outlinks are regarded as feedback to modify blocks´ relevancy. The experiment on Web crawler shows that the new algorithm has some effect on resolving the problem of topic-drift.