Dynamic Metadata Management for Scalable Stream Processing Systems

Adaptive query processing is of utmost importance for the scalability of data stream processing systems due to the long-running queries and fluctuating stream characteristics. An essential prerequisite for adaptive runtime components is the presence of suitable metadata capturing the runtime state. As most of the metadata in such a system gets outdated over time, appropriate update mechanisms are required. Dynamic metadata management deals with the dynamic provision and continuous maintenance of metadata. This paper does not only address the issues in dynamic metadata management such as metadata dependencies and metadata update concepts, but also presents a scalable framework to efficiently manage the diversity of dynamic metadata in todays data stream processing systems. The core of our field-tested metadata framework is a publish-subscribe architecture that enables the system to identify and compute only the currently inquired metadata. This tailored metadata provision is crucial to scalability as maintaining all available metadata at runtime causes significant computational overhead when the number of continuous queries increases.