A Crawler for Local Search

Vertical search engines enable users to find information related to a certain topic. A local search engine is a vertical search engine whose topic revolves around a certain geographical area (such as a city, state, country, etc…) In this paper we describe our experiences developing a crawler for a local search engine for the city of Bellingham, Washington, USA. We focus on the tasks of crawling and indexing a large amount of highly relevant Web pages, and then demonstrate ways in which our search engine has the capability to outperform an industrial search engine.