An Entropy-Based Data Summarization Algorithm in Data Stream System

Recently, there has been much interest in building stream processing applications. In these typical applications, also named data stream applications, data are usually unbounded, continuous, huge in amount, fast arriving, time various and bursting. In order to process the input data stream with real time constraints, overloaded data should be dropped. It is a key problem that how to drop the overloaded data. Through predicting the data which will stream into the system, data summarization algorithm can provide heuristic information to the data stream processing system to drop overloaded input data. In this paper, an entropy-based data summarization algorithm (EBDS) is presented. EBDS is designed to produce samples that are "close" to the whole data. By calculating the entropy of the data in the jumping window, it can get a high predictive accuracy. The experiments indicate that the entropy-based data summarization algorithm has a high predictive accuracy.