Think-aloud protocols: Analyzing three different think-aloud protocols with counts of verbalized frustrations in a usability study of an information-rich Web site

We describe an empirical, between-subjects study on the use of think-aloud protocols in usability testing of an information-rich Web site. This double-blind study used three different types of think-aloud protocols: a traditional protocol, a speech-communication protocol, and a coaching protocol. A silent condition served as the control. Eighty participants were recruited and randomly pre-assigned to one of four conditions. With the goal of keeping unintended bias to a minimum, data analysis did not count the number of identified usability problems by condition, which was considered too subjective. Rather, the study collected the number of verbalized and non-verbalized counts of frustration by condition that users experienced. The study also did a count of the number of verbalized and non-verbalized instances of positive comments by condition that users expressed. Results show that there were no statistical differences in the number of counts by condition with respect to the traditional, speech communication, or coaching condition. The study concludes that simply counting the verbalizations of users by condition does not give enough information to determine whether any of the conditions would lead to a better understanding of the usability problems associated with the Web site.