Modern HPC cluster virtualization using KVM and palacios

In this paper we explore the potential of virtualization being applied to High Performance Computing (HPC). We demonstrate the importance of proper NUMA architecture emulation when running HPC task inside virtual machines on multiple NUMA hosts. We assess KVM/QEMU and Palacios hypervisors and, with proper tuning of hypervisor (including NUMA emulation), we reduce the performance degradation from 10-60% to 1-5% on many tests from HPC Challenge and NAS Parallel Benchmark suites. All tests are performed on modern HPC cluster with high-speed Infiniband interconnect. The cluster nodes are 2-socket 12-core systems, up to 8 nodes were used for computation. Comparing KVM/QEMU and Palacios hypervisors, we conclude that in general the results with NUMA emulation enabled are similar, with KVM providing more stable and predictable results while Palacios being much better on fine-grained tests at a large scale, but showing abnormal performance degradation on a few tests. We believe that the main advantage of Palacios with respect to performance is the reduced amount of noise generated by the virtualization system. This advantage is getting more important when the scale of the system grows.