Adaptive task scheduling in storm

Processing of stream data attracts more and more attention of many big companies and organizations. Storm is a well-known distributed stream processing system that is often used for real-time analysis, online machine learning, continuous computing, distributed remote process call (RPC), etc. In this paper, we study the default scheduler of Storm and other implementations of customized scheduler to discover the primary factors affecting the performance of the cluster. Then, we design and implement an adaptive task scheduler by adding load tracker to monitor the runtime status of the cluster and applying static and dynamic scheduling strategies. At last, we conduct experiments to assess our work by measuring average processing time, overall throughput and stability of the cluster through network bounded and CPU bounded benchmarks. As for average processing time of topologies, the adaptive scheduler achieves about 67% and 30% improvement on cluster of heavy and light load respectively.