A Workload-Driven Approach to Dynamic Data Balancing in MongoDB

With rapid growth of demand on Big Data storage, MongoDB has been a prevalent choice to store unstructured data in recent years. MongoDB evenly distributes data across shard servers to ensure that all the shard servers hold approximately same amount of data and the data access workload will be balanced across these servers. This approach, however, can hardly guarantee the performance of data access when there are hotspots in data because it supposes all the data will be accessed in same patterns. This paper puts forward a workload-driven approach to dynamic data balancing in MongoDB. In this approach, the workload will be monitored in real time by parsing and analyzing log of MongoDB in order to find the hotspots of data. Then, the heat of hotspots will be diffused across shard servers by dynamic data migration. After migration, the workload will be rebalanced across shard servers though data are probably not even distributed, and the performance of hotspots access will be improved. The evaluation of this approach has shown its effectiveness from three aspects, including the improvement of performance, the data migration and the improvement of utilization of CPU.