Title :
Variations in Performance and Scalability: An Experimental Study in IaaS Clouds Using Multi-Tier Workloads
Author :
Jayasinghe, Danushka ; Malkowski, Simon ; Li, Jie ; Qingyang Wang ; Zhikui Wang ; Pu, Calton
Author_Institution :
Coll. of Comput., Georgia Inst. of Technol., Atlanta, GA, USA
Abstract :
The increasing popularity of clouds drives researchers to find answers to a large variety of new and challenging questions. Through extensive experimental measurements, we show variance in performance and scalability of clouds for two non-trivial scenarios. In the first scenario, we target the public Infrastructure as a Service (IaaS) clouds, and study the case when a multi-tier application is migrated from a traditional datacenter to one of the three IaaS clouds. To validate our findings in the first scenario, we conduct similar study with three private clouds built using three mainstream hypervisors. We used the RUBBoS benchmark application and compared its performance and scalability when hosted in Amazon EC2, Open Cirrus, and Emulab. Our results show that a best-performing configuration in one cloud can become the worst-performing configuration in another cloud. Subsequently, we identified several system level bottlenecks such as high context switching and network driver processing overheads that degraded the performance. We experimentally evaluate concrete alternative approaches as practical solutions to address these problems. We then built the three private clouds using a commercial hypervisor (CVM), Xen, and KVM respectively and evaluated performance characteristics using both RUBBoS and Cloudstone benchmark applications. The three clouds show significant performance variations; for instance, Xen outperforms CVM by 75 percent on the read-write RUBBoS workload and CVM outperforms Xen by over 10 percent on the Cloudstone workload. These observed problems were confirmed at a finer granularity through micro-benchmark experiments that measure component performance directly.
Keywords :
benchmark testing; cloud computing; software performance evaluation; Amazon EC2; Cloudstone benchmark applications; Emulab; IaaS cloud performance; IaaS cloud scalability; Open Cirrus; RUBBoS benchmark application; Xen outperforms CVM; experimental measurements; microbenchmark experiments; multitier application; multitier workloads; network driver processing overheads; public Infrastructure as a Service clouds; read-write RUBBoS workload; Benchmark testing; Cloud computing; Clouds; Hardware; Scalability; Virtual machine monitors; Benchmarking; EC2; Emulab; IaaS; RUBBoS; clouds; multi-tier; open cirrus; performance; scalability;
Journal_Title :
Services Computing, IEEE Transactions on
DOI :
10.1109/TSC.2013.46