Online and on-demand partitioning of streaming graphs

Many applications generate data that naturally leads to a graph representation for its modeling and analysis. A common approach to address the size and complexity of these graphs is to split them across a number of partitions, in a way that computations on them can be performed mostly locally and in parallel in the resulting partitions. In this work, we present a framework that enables partitioning of evolving graphs whose elements (nodes and edges) are streamed in an arbitrary order. At a core of our techniques lies a Condensed Spanning Tree (CST) structure that summarizes the graph stream and permits computation of high-quality graph partitions both online and on-demand, without the need to ever look at the whole graph. The partitioning algorithm we present manages to create partitions from streaming graphs with low memory usage, but can also adapt partitions overtime based on different application needs such as minimizing cross-partition edges, balancing load across partitions, elastically adapting partitions based on a maximum load threshold and reducing migration cost. Our experiments with many different real and synthetic graphs demonstrate that our techniques manage to process and partition efficiently millions of graph nodes per second and also adapt them based on different requirements using only the information kept in the compressed CST structure, which can reduce the input graph size down to 1.6%.