Functional testing of feature model analysis tools: a test suite

A feature model is a compact representation of all the products of a software product line. Automated analysis of feature models is rapidly gaining importance: new operations of analysis have been proposed, new tools have been developed to support those operations and different logical paradigms and algorithms have been proposed to perform them. Implementing operations is a complex task that easily leads to errors in analysis solutions. In this context, the lack of specific testing mechanisms is becoming a major obstacle hindering the development of tools and affecting their quality and reliability. In this article, the authors present FaMa test suite, a set of implementation-independent test cases to validate the functionality of feature model analysis tools. This is an efficient and handy mechanism to assist in the development of tools, detecting faults and improving their quality. In order to show the effectiveness of their proposal, the authors evaluated the suite using mutation testing as well as real faults and tools. Their results are promising and directly applicable in the testing of analysis solutions. The authors intend this work to be a first step towards the development of a widely accepted test suite to support functional testing in the community of automated analysis of feature models.