Scalable Real-Time Monitoring for Distributed Applications

In order to assess service quality of a networked application (such as a streaming session), distributed monitoring servers need to continuously collect application-specific performance metrics in real time. Much of the previous work to address this is to use distributed aggregation tree (DAT) rooted at each monitor. However, this approach often leads to high monitoring delay and network stress. In this paper, we study a highly scalable monitoring network for distributed applications. In the network, there are distributed monitors collecting application performance in two steps: first, client applications report their performance to some proxies by means of a client overlay, and then the proxies report the performance to the distributed monitors using another proxy overlay. We first formulate the problem to construct overlays minimizing monitoring delay. The problem is shown to be NP-hard. Then, we present a simple, efficient, and scalable monitoring algorithm called SMon, which continuously reduces network diameter in real time in a distributed manner. Through simulations and actual experimental measurements with implementation, we show that SMon achieves low monitoring delay, network stress, and protocol overhead for distributed applications.