Automatic evaluation of search engines: Using webpages´ content, web graph link structure and websites´ popularity

Measurement of the information retrieval effectiveness of web search engines is an important problem because it can determine which of the available search engines can provide better results to its users. But because of the need of human assessors and their judgments this Measurement is expensive and time consuming. So for solving this problem we introduced an automatic method that is applicable quickly and provides a ranking on the web search engines that were under evaluation without the need of having relevance judgments. In the experiments that were conducted we compared our method with the one proposed by Can et al. and the results acquired from human assessors´ evaluations. The comparisons showed that the introduced method provides rankings that are consistent with the rankings resulted from human assessors´ evaluations and are better than the ones achieved by the Can et al.´s method. Also it is shown that the consistencies observed are statistically significant, so the proposed method can be used for evaluation of web search engines in real environments.