Yadoop: An Elastic Resource Management Solution of YARN

Apache Hadoop YARN is the next generation MapReduce, which is a data-computation framework consisting of the Resource Manager (RM) and per-node slave, the NodeManager (NM). YARN provides explicit support for programming model diversity, so multiple frameworks such as spark, storm can run as applications on YARN. Applications need to apply to Resource Manager for containers to run tasks. That is, applications should specify the amount of CPU and memory they need for each task. To avoid out-of-memory error, applications often apply more resources than they need. This makes resource utilization ratio low. In order to make better use of hardware resources and improve cluster efficiency, we present an elastic resource management approach, which can dynamically expand or shrink container´s size to meet the actual resource needs of tasks. Our experimental results show that the approach can improve the performance of CPU intensive applications significantly by up to 1.5 times for single jobs and 1.3 times for multiple jobs and significantly improve the resource utilization.