Distributed multi-layered workload synthesis for testing stream processing systems

Testing and benchmarking of stream processing systems requires workload representative of real world scenarios with myriad of users, interacting through different applications over different modalities with different underlying protocols. The workload should have realistic volumetric and contextual statistics at different levels: user level, application level, packet level etc. Further realistic workload is inherently distributed in nature. We present a scalable framework for synthesis of distributed workload based on identifying different layers of workload corresponding to different time-scales. The architecture is extensible and modular, promotes reuse of libraries at different layers and offers the flexibility to add additional plug-ins at different layers without sacrificing the efficiency.