Evaluating Performance of Recommender Systems: An Experimental Comparison

Much early evaluation work focused specifically on the "accuracy" of recommendation algorithms. Good recommendation (in terms of accuracy) has, however, to be coupled with other considerations. This work suggests measures aiming at evaluating other aspects than accuracy of recommendation algorithms. Other considerations include (1) coverage, which measures the percentage of a data set that a recommender system is able to provide recommendation for, (2) confidence metrics that can help users make more effective decisions, (3) computing time, which measures how quickly an algorithm can produce good recommendations, (4) novelty/serendipity, which measure whether a recommendation is original, and (5) robustness which measure the ability of the algorithm to make good predictions in the presence of noisy or sparse data. Six collaborative recommendation methods are investigated. Results on artificial data sets (for robustness) or on the real MovieLens data set (for accuracy, novelty, and computing time) are included and analyzed, showing that kernel-based algorithms provide the best results overall.