Greft: Arbitrary Fault-Tolerant Distributed Graph Processing

Many large-scale computing problems can be modeled as graphs. Example areas include the web, social networks, and biological systems. The increasing sizes of datasets has led to the creation of various distributed large scale graph processing systems, e.g., Google Pregel. Although these systems tolerate crash faults, the literature suggests they are vulnerable to a wider range of accidental arbitrary faults (also called Byzantine faults). In this paper we present an algorithm and a prototype of a distributed large-scale graph processing system that can tolerate arbitrary faults. The prototype is based on GPS, an open source implementation of Pregel. Experimental results of the prototype in Amazon AWS are presented, showing that it uses only twice the resources of the original implementation, instead of 3-4 times as usual in Byzantine fault-tolerant systems. This cost may be acceptable for critical applications that require this level of fault tolerance.