A Memory Efficient Approach for Crawling Language Specific Web: The Arabic Web as a Case Study

Web crawlers represent a significant component in Web search engines. They are responsible for making a local copy of Web pages and keeping this local copy up-to-date by periodically refreshing these pages. The decision to refresh a Web page is a tradeoff between the resource utilization and the freshness of the page content. There are various policies as to when to perform a page refresh. A major factor that determines the refresh policy is the change rate of a Web page. In this paper we address the problem of page refresh for the Arabic Web. We present a novel approach that improves the re-crawl scheduling. The proposed technique modifies the information longevity approach to be more suitable for Arabic Web pages. This is done by extracting the Arabic content, and excluding stop list and redundancies that might not contribute significantly to the meaning. This technique saves the scarce memory space in a semantic Arabic Web search engine.