An Online Method for Minimizing Network Monitoring Overhead

Network monitoring is an essential component of network operation and, as the network size increases, it usually generates a significant overhead in large scale networks such as sensor and data center networks. In this paper, we show that measurement correlation often exhibited in real networks can be successfully exploited to reduce the network monitoring overhead. In particular, we propose an online adaptive measurement technique with which a subset of nodes are dynamically chosen as monitors while the measurements of the remaining nodes are estimated using the computed correlations. We propose an estimation framework based on jointly Gaussian distributed random variables, and formulate an optimization problem to select the monitors which minimize the estimation error under a total cost constraint. We show that the problem is NP-Hard and propose three efficient heuristics. In order to apply our framework to real-world networks, in which measurement distribution and correlation may significantly change over time, we also develop a learning based approach that automatically switches between learning and estimation phases using a change detection algorithm. Simulations carried out on two real traces from sensor networks and data centers show that our algorithms outperforms previous solutions based on compressed sensing and it is able to reduce the monitoring overhead by 50% while incurring a low estimation error. The results further demonstrate that applying the change detection algorithm reduces the estimation error up to two orders of magnitude.