NUMA-aware inter-VM communication mechanism in high performance computing

Abstract

Virtualization is technique that virtualizes physical resources existing in a single physical machine and allocate resource to virtual machine. Virtualization is widely adopted in high performance computing environment such as clouding computer center because it has several advantages such as better resource utilization, flexible resource management, fault isolation and supporting for heterogeneous guest OS.

MN-MATE is platform for efficient resource management in a system consists of manycore and non-volatile memory using virtualization technique. The target application of MN-MATE is scientific computation workload and simulation which are highly communication-intensive. Therefore, it is very important to improve inter-VM communication performance because inefficient communication performance can be bottleneck in MN-MATE platform.

However, inefficient inter-VM communication performance can be bottleneck in high performance computing environment because of complex communication path and unnecessary packaging stack. To overcome such limitation, several researches have been proposed based on the fact that virtual machines share same physical hardware. Previous works mainly focus on eliminating complex path between two virtual machines during communication and replace it with direct memory copy using shared buffer between two VMs. These approaches are effective when system has limited number of resource, however scalability problem can occurs if number of resources increase.

Recent HPC architecture such as Intel Nehalem adopt NUMA(Non-Uniform Memory Access) architecture to solve scalability problem when CPU accesses the memory. Instead of sharing a memory controller in whole physical machine, each node has its own local memory and memory controller to solve scalability problem.

However, in NUMA architecture, communication performance can be changed by different shared memory position and memory contention overhead. If CPU and memory are placed in different no...