A Hierarchical Cache Scheme for the Large-scale Web Search Engine

Over the past decade, much research has been done to solve technical challenges regarding the Web search engine, such as crawling Web documents, high performance indexes, and ranking systems using hyperlink analysis. However, implementation details of its query processing system are rarely dealt with in the literature. In this paper we present a distributed architecture for the query processing system and its hierarchal cache scheme. Our paper is based on the development experience of a commercial Web search engine designed to answer 5 million user queries against over 6.5 million Web pages per day. Using the hierarchal cache scheme, we keep a portion of query results in multi-level caches so that excessive I/O or CPU time is not used for query processing. With that scheme, it is possible to reduce around 70% of the server costs.