Direct device-to-device transfer protocol: A new look at the benefits of a decentralized I/O model

Current I/O devices communicate based on the PCIe protocol, and by default, all the traffic passes through the CPU-memory complex. However, this approach causes bottleneck in system throughput, which increases latency and power as the CPU processes device specific protocols to move data between I/O devices. This paper examines the cost of this centralized I/O approach and proposes a new method to perform direct device-to-device I/O communication. Our proof-of-concept implementation using NetFPGA shows that latency can be reduced by more than 2×, CPU utilization can be reduced by up to 18%, and CPU power can be decreased by up to 31 W.