Dynamic data rebalancing in Hadoop

Author

Ashwin Kumar, T.K. ; Jongyeop Kim ; George, K.M. ; Park, Nahea

Author_Institution

Dept. of Comput. Sci., Oklahoma State Univ., Stillwater, OK, USA

fYear

2014

fDate

4-6 June 2014

Firstpage

315

Lastpage

320

Abstract

Current implementation of Hadoop is based on an assumption that all the nodes in a Hadoop cluster are homogenous. Data in a Hadoop cluster is split into blocks and are replicated based on the replication factor. Service time for jobs that accesses data stored in Hadoop considerably increases when the number of jobs is greater than the number of copies of data and when the nodes in Hadoop cluster differ much in their processing capabilities. This paper addresses dynamic data rebalancing in a heterogeneous Hadoop cluster. Data rebalancing is done by replicating data dynamically with minimum data movement cost based on the number of incoming parallel mapreduce jobs. Our experiments indicate that as a result of dynamic data rebalancing service time of mapreduce jobs were reduced by over 30% and resource utilization is increased by over 50% when compared against Hadoop.

Keywords

parallel processing; pattern clustering; replicated databases; resource allocation; Hadoop cluster data; data access; data replication; dynamic data rebalancing service time; heterogeneous Hadoop cluster; jobs service time; minimum data movement cost; parallel mapreduce jobs; replication factor; resource utilization; Clustering algorithms; Dynamic scheduling; Equations; Heuristic algorithms; Load management; Mathematical model; Resource management; Dynamic Data Rebalancing; Hadoop; Replication; heterogeneity; service time; waiting time;

fLanguage

English

Publisher

ieee

Conference_Titel

Computer and Information Science (ICIS), 2014 IEEE/ACIS 13th International Conference on

Conference_Location

Taiyuan

Type

conf

DOI

10.1109/ICIS.2014.6912153

Filename

6912153