A quantitative analysis of high performance computing with Amazon´s EC2 infrastructure: The death of the local cluster?

The introduction of affordable infrastructure on demand, specifically Amazon´s Elastic Compute Cloud (EC2), has had a significant impact in the business IT community and provides reasonable and attractive alternatives to locally-owned infrastructure. For scientific computation however, the viability of EC2 has come into question due to its use of virtualization and network shaping and the performance impacts of both. Several works have shown that EC2 cannot compete with a dedicated HPC cluster utilizing high-performance interconnects, but how does EC2 compare with smaller departmental and lab-sized commodity clusters that are often the primary computational resource for scientists? To answer that question we have run MPI and memory bandwidth benchmarks on EC2 clusters with each of the 64-bit instance types to compare the performance of a 16 node cluster of each to a dedicated locally-owned commodity cluster based on Gigabit Ethernet. Our results show that while EC2 does experience reduced performance, it is still viable for smaller-scale applications.