Scalable and Adaptable Distributed Stream Processing

In this paper we introduce a new architectural design of a large scale distributed stream processing system. The system adopts a two layer architecture. Based on the locality and the natural administrative dependency of the processors, the processors are naturally partitioned into multiple independent entities. Processors within each entity compose the first layer while all the entities comprise the second one. Tightly coupled cooperation is employed within each entity due to the availability of central administration and their close locality. On the contrary, the entities cooperate with each other in a loosely coupled way. Challenges are identified in each layer of the architecture and techniques are proposed to solve them.