A Mixed Coding Scheme for Inverted File Index Compression

Cluster property of d-gaps in inverted lists provides valuable information for improving index compression and subsequently improving the performance of Web search engines. By clustering d-gaps of an inverted list strictly based on a threshold, and then encoding clustered and non-clustered d-gaps using different methods, we can tailor to the specific properties of different d-gaps and achieve better compression ratio. Based on this idea, in this paper we propose a cluster based mixed coding scheme for inverted file index compression: mixed gamma/flat binary code. Experiment results show that the new coding scheme achieves at least equal performance in terms of compression ratio comparing to interpolative code which is considered as one of the most efficient bitwise codes at present. By adjusting the parameters for the new code, even better results can be achieved. Besides, the new code has lower complexity comparing to interpolative code and therefore enables faster encoding and decoding