Diversity-Based Load Shedding Strategy over Pattern Streams

A strategy of finding surprising patterns over data stream without prior knowledge about surprising patterns requires comparing the newly arrived pattern with all kinds of patterns which have emerged. It means all kinds of patterns which have emerged should be stored in memory for the following comparison to ensure real-time response. The patterns needed to be stored in memory are potentially unbounded in size. But the memory resource is limited. To deal with the limited memory problem, we propose a strategy called diversity-based load shedding strategy in this paper. This strategy sheds load with the granularity of pattern and aims to maximize diversity. The experiments on real datasets containing millions of data items demonstrate the feasibility and effectiveness of the proposed strategy.