A Dynamic Load Balancing Technique of Distributed Stream Processing System

Recently, there has been much interest in building stream processing applications. In these typical applications, data are usually unbounded, continuous, huge in amount, fast arriving, time various and out bursting. Delays in data stream processing are very difficult to control because of the outbursting of data incoming and the unpredictable pattern of resource consumption. In order to implement the load balancing, many kinds of methods are adopted such as operator distribution or redistribution, operator migration and load-shedding etc. In this paper, a scalable, efficient, dynamic load balancing mechanisms for distributed data stream processing system, vRing, is presented. It is a hierarchical DHT-based proximity network, and can be deployed on heterogeneous environment. It takes advantage of the scalability and robustness of DHT and the proximity information of network to achieve load balancing. The experiments indicate that vRing can achieve higher efficient performance in data stream processing than FIN, chord and simple-ring.