Use of invariant properties to evaluate the results of fault-injection-based robustness testing of protocol implementations

Robustness testing has as main objective to determine how a system behaves in the presence of unexpected inputs or stressful environmental conditions. An approach commonly used for that purpose is fault injection, in which faults are deliberately injected into a system to observe its behavior. One main limitation of this approach is results evaluation: a system is considered as robust if it does not crash or hang during testing. This is not enough because a system can still continue to execute, but present a wrong behavior. To overcome this limitation, we propose a passive approach for robustness testing, in which the system under test is instrumented for fault injection during runtime, as well as for monitoring its behavior. At the end, the readouts collected are analyzed to determine whether the observed behavior under faults is consistent with properties based on a finite state model of the system. We illustrate the approach using an implementation of the wireless application protocol (WAP). The approach was implemented using off-the-shelf tools; results obtained thus far are presented.