Architecture for Parallel Crawling and Algorithm for Change Detection in Web Pages

In this paper, we put forward a technique for parallel crawling of the Web. The World Wide Web today is growing at a phenomenal rate. The size of the Web as on February 2007 stands at around 29 billion pages. One of the most important uses of crawling the Web is for indexing purposes and keeping Web pages up-to-date, later used by search engine to serve the end user queries. The paper puts forward an architecture built on the lines of a client server architecture. It discusses a fresh approach for parallel crawling the Web using multiple machines and integrates the trivial issues of crawling also. A major part of the Web is dynamic and hence, a need arises to constantly update the changed Web pages. We have used a three-step algorithm for page refreshment. This checks for whether the structure of a Web page has been changed or not, the text content has been altered or whether an image is changed. For the server we have discussed a unique method for distribution of URLs to client machines after determination of their priority index. Also a minor variation to the method of prioritizing URLs on the basis of forward link count has been discussed to accommodate the purpose of frequency of update.