Developing a Web Crawler for Massive Mobile Search Services

Web crawlers collect Web content on the Internet and index them to be retrieved when demanded by a user query. To provide usefulWeb search services, we need a quick crawler that can collect and index the massive amount of content continually accumulating in an efficient way. We introduce a new Web crawler that collects Web content suitable for viewing on mobile terminals such as PDA or cell phones. Moreover, we describe "Mobile Search Service" that provides content suitable for mobile terminals. As of the beginning of 2006, the service offers tens of millions of mobile content entries. In this paper we present the system architecture of our crawler and its performance in actual service.