Abstract :
This paper arises out of a research study into the online help facilities provided
in popular software applications such as word processors. Its particular focus
is on experimental methods of evaluating the effectiveness and usability of
those facilities.
Focus groups, questionnaires, and online surveys had already been used in
other phases of the study, but it was judged that these approaches would be
unsuitable for measuring effectiveness and usability because they are
susceptible to respondents’ subjectivity.
Direct observation of people working on set word-processing tasks was ruled
out initially because of a lack of trained observers; it would have taken too long
for the investigator to observe a large enough sample by himself. Automatic
recording of users’ actions was also rejected, as it would have demanded
equipment and/or software that was not available and seemed too expensive
to acquire.
The approach and techniques described here were an attempt to overcome
these difficulties by using observers drawn from the same population of
students that provided the test subjects; as a by-product, this may also have
enhanced the acceptability (and hence possibly the validity) of the experiments
by reducing the “exam pressure” perceived by participants.
