Title :
A dynamic reconfiguration model for a distributed web crawling system
Author :
Hongfei, Yan ; Jianyong, Wang ; Xiaoming, Li
Author_Institution :
Dept. of Comput. Sci. & Technol., Peking Univ., Beijing, China
Abstract :
A web crawling system employing a parallel and distributed architecture needs to have a mechanism to bring the whole system in a coordinated state when the nodes are added to or removed from the system. This paper presents an efficient dynamic reconfiguration model that can be used in such a system. The study shows that this model leads to some nice properties, such as load balance and low traffic in the system, which contribute to high performance. Currently this model is being implemented in WebGather, a well-known Chinese and English web search engine
Keywords :
Internet; information resources; parallel architectures; search engines; Chinese and English web search engine; WebGather; distributed architecture; distributed web crawling system; dynamic reconfiguration model; load balance; parallel architecture; Computer architecture; Computer networks; Computer science; Crawlers; Laboratories; Search engines; Service oriented architecture; Telecommunication traffic; Traffic control; Web search;
Conference_Titel :
Computer Networks and Mobile Computing, 2001. Proceedings. 2001 International Conference on
Conference_Location :
Los Alamitos, CA
Print_ISBN :
0-7695-1381-6
DOI :
10.1109/ICCNMC.2001.962591
