Finding Problems: When Digital Library Users Act as Usability Evaluators

Users in digital library usability evaluation typically participate as subjects in studies designed and conducted by usability experts and digital library researchers. What happens however when users take the role of the researchers, and with some basic HCI training, design and conduct their own evaluation of a digital library? For several years, teams of students in master´s level HCI classes at Drexel University were given the assignment of designing and carrying out heuristic evaluations of the interface of the Internet Public Library. Their final evaluation reports regularly focused on what, to a usability expert, would not be considered interface issues, such as problems with finding resources in the library. These outcomes contrasted with those of a parallel evaluation of the IPL carried out by doctoral students with a background in HCI, which found interface issues to be the main concerns. A post hoc comparison and analysis of these evaluations highlights differences between users´ and evaluators´ perceptions of usability, and has implications for the design of digital library evaluation and the roles of users and evaluators in such evaluation.