A fast sketch for aggregate queries over high-speed network traffic

There have been security problems and network failures that are hard to resolve, for example, botnets, polymorphic worm/virus, DDoS, etc. To address them, we need to monitor the traffic dynamics and have a network-wide view about them, and more importantly, be able to detect attacks and failures in a timely manner. Due to the rapid increase in the traffic volume, it is often infeasible to monitor every individual flow in the backbone network due to space and time constraints. Instead, we are often required to aggregate packets into a small number of flows and develop the detection methods with aggregated flows, namely aggregate queries. Although it enables ISPs to detect network problems in a timely manner, the flow aggregation cannot preserve certain critical information in network traffic, e.g., IP addresses, port numbers, etc. Due to such missing information, it becomes very difficult (or often infeasible) for ISPs to identify the sources of network attacks or the causes of traffic anomalies, which are important to resolve the network problems effectively. In this paper, we propose an efficient data structure, namely the fast sketch, which can both aggregate packets into a small number of flows, and further enable ISPs to identify the anomalous keys (IP addresses, port numbers, etc.), with small space and time. With it, the number of aggregated flows can achieve the lower bound of the heavy-change detection, i.e., Ω(k log(n/k)), where n is the range of flow keys and k is an upper bound of the number of anomalous keys. In addition, our sketch combines both the combinatorial group testing and the quotient technique to identify anomalous keys, which can guarantee a sub-linear running time. We expect our work will improve the practice for real-time traffic monitoring in a high-speed networked system.