Title :
Predictive dynamic load balancing of parallel hash-joins over heterogeneous processors in the presence of data skew
Author :
Dewan, Hasanat M. ; Mok, Kui W. ; Hernandez, Mauricio ; Stolfo, Salvatore J.
Author_Institution :
Dept. of Comput. Sci., Columbia Univ., New York, NY, USA
Abstract :
We present new algorithms to balance the computation of parallel hash joins over heterogeneous processors in the presence of data skew and external loads. Heterogeneity in our model consists of disparate computing elements, as well as general purpose computing ensembles that are subject to external loading. Data skew appears as significant nonuniformities in the distribution of attribute values of underlying relations that are involved in a join. We develop cost models and predictive dynamic load balancing protocols to detect imbalance during the computation of a single large join. Our algorithms can account for imbalance due to dates skew as well as heterogeneity in the computing environment. Significant performance gains are reported for a wide range of test cases on a prototype implementation of the system
Keywords :
distributed databases; file organisation; parallel processing; performance evaluation; relational databases; resource allocation; attribute values; computation balance; cost models; data skew; disparate computing elements; external loads; general purpose computing; heterogeneous processors; load balancing protocols; parallel hash-joins; performance gains; predictive dynamic load balancing; prototype implementation; single large join; Computer science; Concurrent computing; Costs; Databases; Load management; Partitioning algorithms; Performance gain; Predictive models; Protocols; System testing;
Conference_Titel :
Parallel and Distributed Information Systems, 1994., Proceedings of the Third International Conference on
Conference_Location :
Austin, TX
Print_ISBN :
0-8186-6400-2
DOI :
10.1109/PDIS.1994.331734
