Test Analysis: Searching for Faults in Tests (N)

Tests are increasingly specified as programs. Expressing tests as code is advantageous in that developers are comfortable writing and running code, and tests can be automated and reused as the software evolves. Tests expressed as code, however, can also contain faults. Some test faults are similar to those found in application code, while others are more subtle, caused by incorrect implementation of testing concepts and processes. These faults may cause a test to fail when it should not, or allow program faults to go undetected. In this work we explore whether lightweight static analyses can be cost-effective in pinpointing patterns associated with faults tests. Our exploration includes a categorization and explanation of test patterns, and their application to 12 open source projects that include over 40K tests. We found that several patterns, detectable through simple and efficient static analyses of just the test code, can detect faults with a low false positive rate, while other patterns would require a more sophisticated and extensive code analysis to be useful.