Which of My Failures are Real? Using Relevance Ranking to Raise True Failures to the Top

GUI reference testing is performed to detect regression errors in a modified or patched GUI software. Test cases are executed on the original and modified GUIs, differences in the states of GUI widgets across versions indicate potential defects. However, various factors (e.g., position, flakiness, resolution) create problems for accurate GUI state collection, leading to spurious state mismatches, and hence false positives, these need to be weeded out manually. In this paper, we show that the problem of false positives is significant, often inundating the tester with a large number of false bug reports, requiring a disproportionate amount of manual effort. We develop an entropy-based approach to rank each GUI widget property, and use it to determine whether a state mismatch (indicative of a bug) is real or a false positive. Our empirical evaluation shows that this ranking helps to percolate real bugs to the top of a set of reported bugs, thereby helping to economize tester time.