Sampling Program Inputs with Mutation Analysis: Going Beyond Combinatorial Interaction Testing

Modern systems tend to be highly configurable. Testing such systems requires selecting test cases from a large input space. Thus, there is a need to systematically sample program inputs in order to reduce the testing effort. In such cases, testing the interactions between program parameters has been identified as an effective way to deal with this problem. In these lines, Combinatorial Interaction Testing (CIT) models the program input interactions and uses this model to select test cases. Going a step further, we apply mutation analysis on the CIT input model to select program test cases. Mutation operates by injecting defects to the program input model and measures the number of defects found by the selected test cases. Experiments performed on four real programs show that measuring the number of model-based defects gives a stronger correlation to code-level faults than measuring the number of the exercised interactions. Therefore, the proposed mutation analysis approach forms a valid and more effective alternative to CIT.