Adaptive method to support effective searching over large-scale web documents

Recently, some researchers have found that the abounding search engines cannot support exploratory search effectively. In such case, it requires the search engines know better about the imprecise queries provided by the end users. Actually, it´s hard for the users to formulate the queries, not alone understand by the engines. However, in our study, we find that the search logs in the web community are significantly beneficial for the exploratory search, because various interests´ communities exist in the large-scale web community. In this paper, we treat the search logs in the distributed search servers as footprints. And with these footprints, this study proposes an adaptive method to support effective searching over large-scale web documents. In the adaptive method, logistic regression with trust region applied in Map-reduce environment is devised, and it processes the large-scale web documents in parallel. Thus it can effectively classify the web documents queried by the dynamic imprecise searching. With these results, the method organizes them with frequency tree which shares redundant contents and records the counting for ranking. Extensive experiments demonstrate the merits of our adaptive method to support the exploratory search.