Evaluating task scheduling in hadoop-based cloud systems

Author

Shengyuan Liu ; Jungang Xu ; Zongzhen Liu ; Xu Liu

Author_Institution

Coll. of Comput. & Control Eng., Univ. of Chinese Acad. of Sci., Beijing, China

fYear

2013

fDate

6-9 Oct. 2013

Firstpage

47

Lastpage

53

Abstract

Nowadays, private clouds are widely used for resource sharing. Hadoop-based clusters are the most popular implementations for private clouds. However, because workload traces are not publicly available, few previous work compares and evaluates different cloud solutions with publicly available benchmarks. In this paper, we use a recently-released Cloud benchmarks suite - CloudRank-D to quantitatively evaluate five different Hadoop task schedulers, including FIFO, capacity, naïve fair sharing, fair sharing with delay, and HOD (Hadoop On Demand) scheduling. Our experiments show that with an appropriate scheduler, the throughput of a private cloud can be improved by 20%.

Keywords

cloud computing; resource allocation; scheduling; CloudRank-D; FIFO; HOD; Hadoop On Demand scheduling; Hadoop task schedulers; Hadoop-based cloud systems; Hadoop-based clusters; capacity; naive fair sharing; private clouds; released Cloud benchmarks suite; resource sharing; task scheduling evaluation; workload traces; Benchmark testing; Cloud computing; Data mining; Data warehouses; Delays; Scheduling; Throughput; Cloud; Evaluation; Hadoop; Taks scheduling;

fLanguage

English

Publisher

ieee

Conference_Titel

Big Data, 2013 IEEE International Conference on

Conference_Location

Silicon Valley, CA

Type

conf

DOI

10.1109/BigData.2013.6691697

Filename

6691697