An industry proof-of-concept demonstration of automated combinatorial test

Studies have found that the largest single cost and schedule component of safety-critical, embedded system development is software rework: locating and fixing software defects found during test. In many such systems these defects are the result of interactions among no more than 6 variables, suggesting that 6-way combinatorial testing would be sufficient to trigger and detect them. The National Institute of Standards and Technology developed an approach to automatically generating, executing, and analyzing such tests. This paper describes an industry proof-of-concept demonstration of automated unit and integration testing using this approach. The goal was to see if it might cost-effectively reduce rework by reducing the number of software defects escaping into system test - if it was adequately accurate, scalable, mature, easy to learn, and easy to use and still was able to achieve the required level of structural coverage. Results were positive - e.g., 2775 test input vectors were generated in 6 seconds, expected outputs were generated in 60 minutes, and executing and analyzing them took 8 minutes. Tests detected all seeded defects and in the proof-of-concept demonstration achieved nearly 100% structural coverage.