Effectiveness of Random Testing of Embedded Systems

Embedded systems like those used in automobiles have two peculiar attributes - they are reactive systems where each reaction is influenced by the current state of the system, and their inputs come from small domains. We hypothesise that, because inputs come from small domains, random testing is likely to cover all values in the domain and hence have an effectiveness comparable to other techniques. We also hypothesise that because of the reactive nature long sequences of interactions will be important for testing effectiveness. To test these hypotheses we conducted three experiments on three pieces of code selected from an automotive application. The first two experiments were designed to compare the effectiveness of randomly generated test cases against test cases that achieve the modified condition decision coverage (MCDC) and also evaluate the impact of length of the test cases on effectiveness. The third experiment compares the effectiveness of handwritten test cases against randomly generated test cases of similar length. Our objective is to help practitioners choose an effective technique to test their systems. Our findings from the limited experiments indicate that random test case generation is as effective as manual test generation at the system level. However, for unit testing test case generation to achieve MCDC coverage is more effective than random generation. Combining unit test cases with system level testing increases effectiveness. Our final observation is that increasing the test case length improves the effectiveness of a test suite both at the unit and system level.