Use of Relevance Criteria across Stages of Document Evaluation: On the Complementarity of Experimental and Naturalistic Studies

This paper provides an overview of empirical investigations of peopleʹs use of relevance criteria. A laboratory experiment and a naturalistic study were conducted to explore the patterns of movement in use of relevance criteria during real-time document evaluation processes. The expectation was that the subjects would apply different frames of reference for relevance decision making as they moved from one stage of the document evaluation to another, and that change would occur not only for particular criteria but also for classes of criteria. The examination of such change was performed at two levels: a micro-level analysis, centering on the use of individual criteria, and a macro-level analysis, concentrating on the use of criteria classes. General results of the two studies are included in this paper. Both projects provided information on criteria patterns, which challenge the conventional meaning-based approach to criteria classification. Building on these findings, a classification scheme and criteria taxonomy that may enable generalization of criteria across different contexts is proposed.