H-GAP: estimating histograms of local variables with accuracy objectives for distributed real-time monitoring

We present H-GAP, a protocol for continuous monitoring, which provides a management station with the value distribution of local variables across the network. The protocol estimates the histogram of local state variables for a given accuracy and with minimal overhead. H-GAP is decentralized and asynchronous to achieve robustness and scalability, and it executes on an overlay interconnecting management processes in network devices. On this overlay, the protocol maintains a spanning tree and updates the histogram through incremental aggregation. The protocol is tunable in the sense that it allows controlling, at runtime, the trade-off between protocol overhead and an accuracy objective. This functionality is realized through dynamic configuration of local filters that control the flow of updates towards the management station. The paper includes an analysis of the problem of histogram aggregation over aggregation trees, a formulation of the global optimization problem, and a distributed solution containing heuristic, tree-based algorithms. Using SUM as an example, we show how general aggregation functions over local variables can be efficiently computed with H-GAP. We evaluate our protocol through simulation using real traces. The results demonstrate the controllability of H-GAP in a selection of scenarios and its efficiency in large-scale networks.