Predictable, Efficient System-Level Fault Tolerance in C^3

Predictable reliability is an increasingly important aspect of embedded and real-time systems. This includes the ability to recover from unknown faults in a manner that maintains system timing guarantees, even when these faults occur within system components. This paper presents the C³ system, which is the first system implementation we know of for predictable, system-level fault tolerance that doesn´t require physical redundancy. We introduce both the system design, and two timing analyses that enable the predictable recovery from faults in operating system components, and identify recovery inversion as a main impediment to schedulable recovery. C³ provides fault-tolerance for low-level system components using a combination of efficient u-reboots, and an interface-driven mechanism to recreate component state. C³ introduces on-demand recovery that properly prioritizes aspects of the recovery process to avoid this inversion and not inhibit system timeliness. We compare this system to both eager recovery, and to check pointing of a Para virtualized real-time OS.