Exploiting Routing Information Encoded into Backlinks to Improve Topical Crawling

Local link analysis of topical graphs on the Web allows to experiment focused crawling strategies in a detailed way. In this scope, models, parameters and metrics used to orientate the crawler can be better understood, tuned and evaluated. We develop a methodological and experimental approach that exploits link analysis in order to determine what constitutes a good content analysis metric able to guide efficiently topical crawlers toward highly relevant areas of the Web. Our experimentations show that partial knowledge of the local topology of topical graph highlights our understanding of routing capabilities of various metrics. Furthermore, our experimentations demonstrate that significant crawling efficiency improvement can be reached.