Low-Latency Linux Drivers for Ethernet over High-Speed Networks

Nowadays, high computing demands are often tackled by clusters of single computers, each of which is basically an assembly of a growing number of CPU cores and main memory, also called a node; these nodes are connected by some kind of communication network. With the growing speed and number of CPU cores, the network becomes a severe bottleneck limiting overall cluster performance. Highspeed interconnects like InfiniBand, SCI, and Dolphin DX are good for alleviating this communication bottleneck, when the communication software is either based on IP or specifically adapted to the interconnect. Software written to communicate directly via Ethernet frames can not be used this way, though. In this article, we present two drivers for Linux that fill this gap. ETHOS is a very generic Ethernet over Sockets driver. With this driver it is possible to use any interconnect that offers a sockets interface as replacement for Ethernet. The second driver, ETHOM, sacrifices the compatibility with a wide range of interconnects in favour of higher performance on top of Dolphinʹs high-speed networks SCI and DX. It enhances their functionality by offering an Ethernet and with that an IP interface. Both drivers allow usage of layer-2 kernel functionality like interface bonding and bridging. By means of various measurements, we show that ETHOS and ETHOM with InfiniBand, SCI, or DX offer a two- to threefold increase in communication performance over Gigabit Ethernet.