Improving Hadoop Monetary Efficiency in the Cloud Using Spot Instances

Infrastructure-as-a-Service (IaaS) cloud providers offer many elasticities and flexibilities for users to run their systems in the cloud. The monetary cost issues of running those systems in the cloud are hardly ignored and there is less work discussing improving the monetary efficiency of running large scale systems in dynamic cloud environments. In this paper, we focus on improving the monetary efficiency of running Hadoop systems in the dynamic public cloud. In particular, we carry out detailed study on improving the monetary efficiency by leveraging spot instances. From a cloud broker´s perspective, we propose a price-aware virtual machine auto-scaling with migration algorithm to improve the monetary efficiency of running Hadoop in the cloud using spot instances. We evaluate our proposed algorithm through simulation using Amazon EC2 spot price traces and real world workload traces. Compared with other baseline algorithms, our approach can improve the monetary efficiency by up to 9.3x.