Dynamic Load Balancing in Data Grids by Global Load Estimation

Peer-to-Peer (P2P) technology can be utilized to combine remote resources and build distributed, high performance database systems, called data grids, which help to handle the rapidly increasing volumes of data produced by disciplines like astrophysics, biology, or geology. One major challenge of data grids are skewed query patterns which cause load imbalances and heavily diminish performance and availability. To avoid hot spots, sophisticated load balancing techniques are required. We present a dynamic replication strategy which prevents hot spots by dynamically replicating the hot data on different locations. The main questions of such a strategy are when to copy which data to what receivers and when to delete the copies. To answer these questions we propose a low-overhead, decentralized method which is able to deliver a highly accurate estimate of the global load and the single peer loads to all clients. We use that information in an optimization problem to determine the data to be replicated and the optimal replica receivers. A simulated performance evaluation based on a real-world scenario demonstrates the effectiveness of the approach.