Title :
Predictive data and energy management in GreenHDFS
Author :
Kaushik, Rini T. ; Abdelzaher, Tarek ; Egashira, Ryota ; Nahrstedt, Klara
Author_Institution :
Univ. of Illinois at Urbana-Champaign, Urbana, IL, USA
Abstract :
The sheer scale and rapid rise of Big Data mandates highly scalable, self-adaptive, and energy-conserving data-intensive compute clusters. Based on our analysis of the traces from a production Hadoop cluster at Yahoo!, we observe that file size, file lifespan, and file heat are statistically correlated and very strongly associated with the hierarchical directory structure (i.e., absolute file path) in which the files are organized. Leveraging that observation, we present predictive GreenHDFS; an energy-conserving variant of the Hadoop distributed file system that uses a supervised machine learning technique to learn the correlation between the directory hierarchy and the file attributes to guide novel predictive file zone placement, migration, and replication policies that significantly outperform the current state-of-the-art reactive approaches. Using real-world traces from a large-scale (2600 servers, 5 Petabytes) production Hadoop cluster at Yahoo! in our GreenHDFS simulations, we show how predictive GreenHDFS results in a much better trade-off between performance and energy consumption.
Keywords :
distributed databases; energy management systems; file organisation; learning (artificial intelligence); network operating systems; GreenHDFS simulations; Hadoop distributed file system; Yahoo; absolute file path; data-intensive compute clusters; directory hierarchy; energy consumption; energy management; file attributes; file heat; file lifespan; file migration; file size; hierarchical directory structure; large-scale production Hadoop cluster; predictive GreenHDFS; predictive data; predictive file zone placement; replication policy; supervised machine learning technique; Facebook; Heating; Training;
Conference_Titel :
Green Computing Conference and Workshops (IGCC), 2011 International
Conference_Location :
Orlando, FL
Print_ISBN :
978-1-4577-1222-7
DOI :
10.1109/IGCC.2011.6008563