Abstract :
This article focuses on the relevance judgments made
by health information users who use the Web. Health
information users were conceptualized as motivated
information users concerned about how an environmental
issue affects their health. Users identified their own
environmental health interests and conducted a Web
search of a particular environmental health Web site.
Users were asked to identify (by highlighting with a
mouse) the criteria they use to assess relevance in both
Web search engine surrogates and full-text Web documents.
Content analysis of document criteria highlighted
by users identified the criteria these users relied on
most often. Key criteria identified included (in order of
frequency of appearance) research, topic, scope, data,
influence, affiliation, Web characteristics, and authority/
person. A power-law distribution of criteria was observed
(a few criteria represented most of the highlighted
regions, with a long tail of occasionally used criteria).
Implications of this work are that information retrieval
(IR) systems should be tailored in terms of users’ tendencies
to rely on certain document criteria, and that
relevance research should combine methods to gather
richer, contextualized data. Metadata for IR systems,
such as that used in search engine surrogates, could be
improved by taking into account actual usage of relevance
criteria. Such metadata should be user-centered
(based on data from users, as in this study) and contextappropriate
(fit to users’ situations and tasks).
