Meta-Evaluation of a Context-Aware Mobile Device Usability

This article analyzes the completeness and the correctness of usability analyses conducted in usability laboratories compared to analyses conducted in reality testing (where the context is real and video recordings impossible for technical or privacy reasons). To do so, we set up a meta-evaluation of a mobile device in a quasi-realistic context. We collected two sets of data. The first type refers to data usually available in a classic usability laboratory, the second simulates data which could be collected in reality testing. We did two independent analyses and then, afterward, compared their findings. The meta- evaluation concludes that it is possible -with limitations- to evaluate the usability of a mobile device in reality testing without video recordings. However, the usability problems detected may remain unexplained without the help of the video recordings.