Hadoop branching: Architectural impacts on energy and performance

Data centers are notorious energy consumers. In fact, studies have shown that for every $1 spent on hardware in the datacenter, $0.50 is spent on powering this hardware over its lifetime. Data centers host real or virtual (i.e., cloud) clusters that often execute large compute jobs using MapReduce, of which Hadoop is a popular implementation. Like other successful open source projects, Hadoop has been maintained and evolved over time with new resource management features being added over time in an effort to improve performance, raising questions as to whether such architectural evolution has achieved its goal, and if so, at what cost. In this work we apply Green Mining to find out that later versions of Hadoop - who exhibit more dynamic resource control - can suffer from serious energy consumption performance regressions.