A Case Study of Automating User Experience-Oriented Performance Testing on Smartphones

We have developed a platform named Advanced Test Environment (ATE) for supporting the design and the automatic execution of UX tests for applications running on Android smartphones. The platform collects objective metrics used to estimate the UX. In this paper, we investigate the extent that the metrics captured by ATE are able to approximate the results that are obtained from UX testing with real human users. Our findings suggest that ATE produces UX estimations that are comparable to those reported by human users. We have also compared ATE with three widespread benchmark tools that are commonly used in the industry, and the results show that ATE outperforms these tools.