Improving Index Compression Using Cluster Information

The clustering property of document collections in Web search engines provides valuable information for improving index compression. By clustering d-gaps of an inverted list and then encoding clustered and non-clustered d-gaps using different codes, we can tailor to the specific properties of different d-gaps and achieve better compression ratio. Further improvement on index compression can be achieved by adoptively adjusting the cluster threshold for inverted lists. Based on these ideas, in this paper we propose adaptive cluster based mixed codes for inverted file index compression. Experiment results show that codes using adaptive cluster based mixed approach have better performance in terms of compression ratio and lower complexity comparing to interpolative code which is considered as one of the most efficient bitwise codes at present