Design and implementation of parallel statiatical algorithm based on Hadoop´s MapReduce model

Author

Duan, Songqing ; Wu, Bin ; Wang, Bai ; Yang, Juan

Author_Institution

Beijing Key Lab. of Intell. Telecommun. Software & Multimedia, Beijing Univ. of Posts & Telecommun., Beijing, China

fYear

2011

fDate

15-17 Sept. 2011

Firstpage

134

Lastpage

138

Abstract

The rapid growth of data promotes the development of parallel computing. MapReduce, which is a simplified programming model of distributed parallel computing, is becoming more and more popular. In this paper, we design and implementation of parallel statistical algorithm based on Hadoop´s MapReduce model. The algorithm, which is used to grasp the overall characteristics of massive data, involves the calculation of central tendency, dispersion and distribution tendency. By experiment, we come to the conclusion that the algorithm is suitable for dealing with large-scale data.

Keywords

parallel algorithms; Hadoop MapReduce model; central tendency calculation; dispersion calculation; distributed parallel computing; distribution tendency calculation; parallel computing; parallel statiatical algorithm; Algorithm design and analysis; Cloud computing; Computational modeling; Dispersion; File systems; Gaussian distribution; Programming; Central Tendency; Dispersion; Distribution Tendency; Hadoop; MapReduce; Parallel Statistical Algorithm;

fLanguage

English

Publisher

ieee

Conference_Titel

Cloud Computing and Intelligence Systems (CCIS), 2011 IEEE International Conference on

Conference_Location

Beijing

Print_ISBN

978-1-61284-203-5

Type

conf

DOI

10.1109/CCIS.2011.6045047

Filename

6045047