Using the Information Seeker to Elicit Construct Models for Search Engine Evaluation

With the rise of interactive search engines, direct access to information is available to the end user. Evaluation of these systems is clearly important for their comparison and development. From the perspective of the user engaged with the system in some information-seeking task, the range of relevant evaluation factors could be considerable. This article presents an investigation into the suitability of repertory grid technique eliciting a mental model of search engines in the user. Such a model comprises constructs held important to the user, which can be used to evaluate the system. The design of the repertory grid application is described and the data analyzed in two stages. The important issues in determining the suitability of the method are identified and the constructs are analyzed to determine their association, discriminatory ability, and clustering around a central overall rating. From this, it is concluded that repertory grid technique is appropriate for user-centered determination of evaluative constructs. Further analysis of the characteristics of the resulting construct set is undertaken towards establishing an underlying user model of the search engines, and further research is identified for its use in system evaluation.