Quantitative similarity-based evaluation of text retrieval algorithms

Text retrieval engines, such as search engines, always return a list of documents in response to a given query. Existing evaluations of text retrieval algorithms mostly use precision and recall of the returned list of documents as main quality measures of a search engine. In this paper, we propose a novel approach for comparing different algorithms adopted by different search engines and evaluate their performance. In our approach, the results of each algorithm is treated as an inter-related set of documents and the effectiveness of the algorithm is evaluated based on the degree of relation in the set of documents. After verifying the correctness of the evaluation measure by examining the results of the two retrieval algorithms, BM25 and pivoted normalization, and comparing these results with an ideal ranking, we compare the results of these algorithms and investigate the impact of certain major factors like stemming on the results of the suggested algorithm. The effectiveness of our proposed method is justified through obtained experimental results.