The Composite Endpoint Protocol (CEP): scalable endpoints for terabit flows

We introduce the Composite Endpoint Protocol (CEP), which efficiently composes a set of transmission elements to support high speed flows which exceed the capabilities of a single computer. CEP´s unique capabilities include: (1) allowing multiple processes (a composite endpoint) to take part in a single logical connection, (2) providing a simple, flexible interface to describe data layouts and composite endpoint communication to user programs, (3) providing efficient transfer scheduling which coordinates heterogeneous nodes to achieve good composite performance, and (4) a scalable architecture which supports large numbers of participants in a composite endpoint. We describe the design of CEP, an initial implementation, and an empirical evaluation exhibiting the above capabilities. We have achieved speeds over 32 Gbps using commodity cluster hardware with linear scalability performance 7x naive approaches, and low overhead.