QUAY: a data stream processing system using chunking

Data stream processing has emerged as a recent research direction focusing on new generation database applications, in which data records from remote source sites flow continuously to a processing site. Queries residing in the processing site are triggered and evaluated upon the arrival of their interested data records. There are two important aspects that distinguish data stream processing systems from conventional database systems. First, the roles of queries and data records are swapped; queries are stationary while data records are dynamic. Query indexing becomes an essential performance determining issue. Second, the expectedly high data flow rate aggravates data index maintenance overheads. To address the problems thus arisen, we propose and develop a data stream processing system called QUAY. We present the design, implementation and evaluation of QUAY. The core technique that we use is "chunking" which clusters and indexes both queries and data records in a unified way as chunks. To process window join operation from stream sources, we propose an adaptive selection-join arrangement for a huge number of selection-join queries to share expensive join operations. Through a set of intensive performance evaluation experiments, we show that the chunking organization, operating under our proposed adaptive selection-join arrangement, yields desirably good performance.