Optimizing the Web Search Engines with Features and Caching

Large web search engines have to answer thousands of queries per second with interactive response times under tight latency constraints. Query processing is a major cost factor in operating large web search engines. To keep up with this immense workload, a number of techniques such as caching, index compression, and index and query pruning are used to improve scalability. We focus on two techniques, inverted index compression and feature-based caching. We perform a comparison and evaluation of several inverted compression algorithms. We then set a new feature-based eviction policies that achieve significant improvements over previous methods. Experimental results shows that this approach can achieve hit rate improvements.