Software tool combining fault masking with user-defined recovery strategies

The voting farm, a tool which implements a distributed software voting mechanism for a number of parallel message passing systems, is described. The tool, developed in the framework of EFTOS (embedded fault tolerant supercomputing), can be used in standalone mode or in conjunction with other EFTOS fault tolerance tools. In the former case, exploitation of the mechanism is described, e.g. to implement restoring organs (N-modular redundancy systems with N-replicated voters); in the latter case, it is shown how it is possible for the user to implement in an easy and effective way a number of different recovery strategies via a custom, high-level language. Combining such strategies with the basic fault masking capabilities of the voting tool makes it possible to set up complex fault tolerant systems such as, for instance, N-and-M-spare systems or gracefully degrading voting farms. The impact that the tool can have on reliability is discussed, and it is shown how, besides structural design goals like fault transparency, the tool achieves replication transparency, a high degree of flexibility and ease-of-use, and good performance