DEFINE: a distributed fault injection and monitoring environment

This paper presents a distributed fault injection and monitoring environment (DEFINE) as a tool to evaluate system dependability, to investigate fault propagation, and to validate fault-tolerant mechanisms. DEFINE can inject both hardware faults (hardware-induced software errors) and software faults into any process running in a distributed system, either in user mode or in supervisor mode, and monitor the fault impact and propagation in software systems and among machines. It employs two fault injection techniques: (i) using hardware clock interrupts to control the time of fault injection and activation, and (ii) using software traps to inject all the faults except communication faults and memory faults in the data/stack segment. Experiments on six Sun SPARCstations to study the system behavior under faults are conducted to demonstrate the application of DEFINE