Automatic New Topic Identification in Search Engine Transaction Logs Using Multiple Linear Regression

Content analysis of search engine user queries is an important task for search engine research, and identification of topic changes within a user search session is a key issue in content analysis of search engine user queries. The purpose of this study is to provide automatic new topic identification of search engine query logs, and estimate the effect of statistical characteristics of search engine queries on new topic identification. By applying multiple linear regression and ANOVA on a sample data log from the FAST search engine, we have reached the following findings: 1) We demonstrated that the statistical characteristics of Web search queries are effective on shifting to a new topic; 2) Multiple linear regression is a successful tool for estimating topic shifts and continuations. This study provides statistical proof for the relationship between the non-semantic characteristics of Web search queries and the occurrence of topic shifts and continuations.