Semantic Mutation Analysis of Floating-Point Comparison

Semantic Mutation Testing (SMT) is a technique that aims to capture errors caused by possible misunderstandings of the semantics of a description language. This paper focuses on the use of SMT to represent possible problems caused by the use of Floating Point Comparison (FPC) since this feature of programming languages can lead to subtle errors. We describe six FPC semantic mutation operators that have been implemented in a C SMT tool. These operators mutate a C program by introducing tolerances using three different algorithms. The paper reports on the results of experiments that explored the proposed mutation operators. It was found that random test suites were not good at killing the resultant mutants, suggesting also that random test suites are poor at revealing FPC problems. We therefore devised a new approach to generate test data to kill these mutants. The manually generated test suites produced using the new approach were more effective in killing FPC mutants. In addition, the random test suites and manual test suites killed different sets of FPC mutants. The impact of using three different algorithms in FPC mutation was also investigated with no dominates relationships being found between the three types of FPC operators. Finally, we ran the same experiments on a computer with a different configuration. We found that slightly different sets of mutants were killed on the two computers, indicating that portability problems can be introduced by FPC.