An Information Measure for Comparing Top k Lists

Comparing the top k elements between two or more ranked results is a common task in many contexts and settings. A few measures have been proposed to compare top k lists with attractive mathematical properties, but they face a number of pitfalls and shortcomings in practice. This work introduces a new measure to compare any two top k lists based on measuring the information these lists convey. Our method investigates the compressibility of the lists, and the length of the message to encode losslessly the lists gives a natural and robust measure of their variability. This information-theoretic measure objectively reconciles all the main considerations that arise when measuring (dis-)similarity between lists: the extent of their non-overlapping elements, the amount of disarray among overlapping elements, the measurement of displacement of actual ranks (positions) of their overlapping elements. We demonstrate that our measure is intuitively simple and superior to other commonly used measures. To the best of our knowledge, this is the first attempt to address the problem using information compression as its basis.