A test automation language framework for behavioral models

Model-based testers design tests in terms of models, such as paths in graphs. This results in abstract tests, which have to be converted to concrete tests because the abstract tests use names and events that exist in the model, but not the implementation. Model elements often appear in many abstract tests, so testers write the same redundant code many times. However, many existing model-based testing techniques are very complicated to use in practice, especially in agile software development. Thus, testers usually have to transform abstract tests to concrete tests by hand. This is time-consuming, labor-intensive, and error-prone. This paper presents a language to automate the creation of mappings from abstract tests to concrete tests. Three issues are addressed: (1) creating mappings and generating test values, (2) transforming graphs and using coverage criteria to generate test paths, and (3) solving constraints and generating concrete tests. Based on the language, we developed a test automation language framework. The paper also presents results from an empirical comparison of testers using the framework with manual mapping on 11 open source and 6 example programs. We found that the automated test generation method took 29.6% of the time the manual method took on average, and the manual tests contained 48 errors in which concrete tests did not match their abstract tests while the automatic tests had zero errors.