Combinatorial Test Design in the TOSCA Testsuite: Lessons Learned and Practical Implications

The advantage of combinatorial techniques over less structured approaches is supported by the experience from numerous real-world projects where a significant reduction of the number of test cases has been achieved without compromising functional coverage. However, to fully benefit from combinatorial testing, the applied techniques and tools have to satisfy the requirements and needs of testers and practitioners. In this paper we explore such requirements distilled from testing software systems for over 15 years across a wide range of projects in business and industry. Their practical implications span from mastering the combinatorial explosion over support for fault localization to understandability, changeability and maintainability. Finally, the paper illustrates how the different combinatorial techniques are able to meet these requirements. The combinatorial techniques discussed in this paper are part of the TOSCA Test suite developed by TRICENTIS.