uStream: A User-Level Stream Protocol over Infiniband

As one of the most popular high speed networks, InfiniBand demonstrates several enhanced features, such as RDMA and zero-copy mechanisms, which offer high bandwidth and low latency. Communication stacks IPoIB and SDP (Sockets Direct Protocol) have been proposed on InfiniBand for sockets based applications to take advantage of these features. However, these protocols are inefficient to utilize the performance capabilities provided by the physical network. In order to fully exploit the high performance of InfiniBand, we present uStream, a relatively simple and efficient protocol on the user-level socket layer with a stream interface to enable RDMA capability and zero-copy mechanism. Experiment results have shown that the performance of uStream is comparable to raw InfiniBand Verbs/RDMA interface, and uStream outperforms SDP with 7.9 ¿s minimum latency and 10.4 Gbps peak bandwidth in our testbed. In addition, a Java communication library, jStream, is built upon uStream to enable Java to use RDMA directly. Preliminary results have shown Java clusters communication performance has been improved to a level similar to C programs over InfiniBand through combined uStream and jStream.