Breaking the connection: RDMA deconstructed

Author

Sivaram, Rajeev ; Govindaraju, Rama K. ; Hochschild, Peter ; Blackmore, Robert ; Chaudhary, Piyush

Author_Institution

Adv. Clustering Technol. Team, IBM Corp., USA

fYear

2005

fDate

17-19 Aug. 2005

Firstpage

36

Lastpage

42

Abstract

The architecture, design and performance of RDMA (remote direct memory access) over the IBM HPS (high performance switch and adapter) are described. Unlike conventional implementations such as InfiniBand, our RDMA transport model is layered on top of an unreliable datagram interface, while leaving the task of enforcing reliability to the ULP (upper layer protocol). We demonstrate that our model allows a single MPI task to deliver bidirectional bandwidth of close to 3.0 GB/s across a single link and 24.0 GB/s when striped across 8 links. In addition, we show that this transport protocol has superior attributes in terms of a) being able to handle RDMA packets coming out of order; b) being able to use multiple routes between a source-destination pair and c) reducing the size of adapter caches.

Keywords

cache storage; message passing; routing protocols; telecommunication switching; transport protocols; 24.0 Gbit/s; 3.0 Gbit/s; IBM HPS; MPI; RDMA; ULP; adapter cache; high performance switch; network routing; remote direct memory access; source-destination pair; transport protocol; unreliable datagram interface; upper layer protocol; Access protocols; Bandwidth; Computer networks; Fabrics; Network interfaces; Out of order; Random access memory; Routing; Switches; Transport protocols;

fLanguage

English

Publisher

ieee

Conference_Titel

High Performance Interconnects, 2005. Proceedings. 13th Symposium on

ISSN

1550-4794

Print_ISBN

0-7695-2449-4

Type

conf

DOI

10.1109/CONECT.2005.9

Filename

1544575