goProbe: a scalable distributed network monitoring solution

The Internet has developed into the primary means of communication, while ensuring availability and stability is becoming an increasingly challenging task. Traffic monitoring enables network operators to comprehend the composition of traffic flowing through individual corporate and private networks, making it essential for planning, reporting and debugging purposes. Classical packet capture and aggregation concepts (e.g. NetFlow) typically rely on centralized collection of traffic metadata. With the proliferation of network enabled devices and the resulting increase in data volume, such approaches suffer from scalability issues, often prohibiting the transfer of raw metadata as such. This paper describes a decentralized approach, eliminating the need for a central collector and storing local views of network traffic patterns on the respective devices performing the capture. In order to allow for the analysis of captured data, queries formulated by analysts are distributed across all devices. Processing takes place in a parallelized fashion on the respective local data. Consequently, instead of continually transferring raw metadata, significantly smaller aggregate results are sent to a central location which are then combined into the requested final result. The proposed system describes a lightweight and scalable monitoring solution, enabling the efficient use of available system resources on the distributed devices, hence allowing for high performance, real-time traffic analysis on a global scale. The solution was implemented and deployed globally on hosts managed and maintained by a large managed network security services provider.