Finding correlated heavy-hitters over data streams

We consider online mining of correlated heavy-hitters (CHH) from network data streams. Given a multidimensional dataset, a correlated aggregate query first filters a subset by applying a predicate along a primary dimension, and then computes aggregates along a secondary dimension of that subset data. We consider queries of the following form: "In a stream of (x, y) tuples, on the subset H of all x values that are heavy-hitters, maintain those y values that occur frequently with the x values in H". This query arises naturally in situations where we need to track not only the identity of frequently occurring elements in a stream, but also additional information associated with these elements along other dimensions. Prior work on tracking heavy-hitters has focused only on tracking the identity and frequency of heavy-hitters on a single dimensional stream, and yield little information about correlated heavy-hitters. Our online data stream algorithm is easy to implement and uses workspace which is orders of magnitude smaller than the stream itself. We present provable guarantees on the maximum error estimates, as well as experimental results, that demonstrate the space-accuracy trade-off on a large data stream of packet headers from a backbone network link.