A Novel Mobile Crawler System Based ‎ on Filtering off Non-Modified Pages for ‎Reducing Load on the Network

The studies in the literature show that about 40% of the current Internet traffic and bandwidth consumption is due to web ‎crawlers that retrieve pages for indexing by the different search engines. This traffic and bandwidth consumption will increase ‎in future due to the exponential growth of the web. This paper addresses the problem of bandwidth consumption by introducing ‎an efficient indexing system based on mobile crawlers. The proposed system employs mobile agents to crawl the pages. These ‎mobile agent based crawlers retrieve the pages, process them, compare their data to filter out pages that are not modified after ‎last crawl, and then compress them before sending them to the search engine for indexing. The experimental results of the ‎proposed system are very encouraging