Probing and fault injection of protocol implementations

Ensuring that a distributed system with strict dependability constraints meets its prescribed specification is a growing challenge that confronts software developers and system engineers. This paper presents a technique for probing and fault injection of fault-tolerant distributed protocols. The proposed technique, called script-driven probing and fault injection, can be used for studying the behavior of distributed systems and for detecting design and implementation errors of fault-tolerant protocols. The focus of this work is on fault injection techniques that can be used to demonstrate three aspects of a target protocol: i) detection of design or implementation errors, ii) identification of violations of protocol specifications, and iii) insight into design decisions made by the implementers. The emphasis of our approach is on experimental techniques intended to identify specific “problems” in a protocol or its implementation rather than the evaluation of system dependability through statistical metrics such as fault coverage. To demonstrate the capabilities of this technique, the paper describes a probing and fault injection tool, called the PFI tool (Probe/Fault Injection Tool), and a summary of several extensive experiments that studied the behavior of two protocols: the transmission control protocol (TCP) and a group membership protocol (GMP)