On the Cost of Tunnel Endpoint Processing in Overlay Virtual Networks

Overlays are increasingly used to carry out virtual networks in multi-tenant data centers. However, the encapsulation layer of these overlay virtual networks increases the processing cost and degrades the system performance when the tunnel endpoints are implemented in software. In this paper, we investigate this processing cost in terms of clock cycles by instrumenting the main processing layers of the Linux VXLAN tunnel endpoint. We found that the VXLAN encapsulation layer adds 21% of CPU cycles to the processing of an MTU-size packet. We also observed that encapsulation overhead translates into a bandwidth performance drop of 31.8% and a latency increase of 32.5%. We propose and emulate a minimal hardware acceleration technique for VXLAN that can bring back the lost performance. The experiment results show that 97.1% of the bandwidth performance and 94.4% of the latency performance can be achieved at constant CPU utilization.