When a GUI Regression Test Failed, What Should be Blamed?

Script-based automated regression testing is widely used in industry. In this work, we focus on failed tests in a real regression test project. The causes of 197 failed tests produced in automated testing are examined and categorized based on an analysis procedure presented. The result shows that incorrect scripts, oracle mismatches, test tool bugs and misconfigurations involved in testing contribute most to failures instead of product bugs. Detecting and fixing the false positives are laborious and time-consuming. Our empirical study demonstrates that the benefits of test automation are obliterated to some extent in practical settings. To improve the effectiveness of automated regression testing further, some issues should receive more attention.