Title :
A storage-centric analysis of MapReduce workloads: File popularity, temporal locality and arrival patterns
Author :
Abad, Cristina L. ; Roberts, Nick ; Yi Lu ; Campbell, Roy H.
Author_Institution :
Univ. of Illinois at Urbana-Champaign, Urbana, IL, USA
Abstract :
A huge increase in data storage and processing requirements has lead to Big Data, for which next generation storage systems are being designed and implemented. However, we have a limited understanding of the workloads of Big Data storage systems. We consider the case of one common type of Big Data storage cluster: a cluster dedicated to supporting a mix of MapReduce jobs. We analyze 6-month traces from two large Hadoop clusters at Yahoo! and characterize the file popularity, temporal locality, and arrival patterns of the workloads. We identify several interesting properties and compare them with previous observations from web and media server workloads. To the best of our knowledge, this is the first study of how MapReduce workloads interact with the storage layer.
Keywords :
Internet; data analysis; file servers; parallel programming; pattern clustering; storage management; Hadoop cluster; MapReduce workload; Web server workload; Yahoo!; arrival patterns; big data storage cluster; data processing requirement; data storage centric analysis; file popularity; media server workload; next generation storage system; temporal locality; Data handling; Data storage systems; Distribution functions; Information management; Media; Servers; Sociology; Big Data; HDFS; MapReduce; access patterns;
Conference_Titel :
Workload Characterization (IISWC), 2012 IEEE International Symposium on
Conference_Location :
La Jolla, CA
Print_ISBN :
978-1-4673-4531-6
DOI :
10.1109/IISWC.2012.6402909
