High performance FPGA-based scatter/gather DMA interface for PCIe

Today, Data Acquisition Systems often use PCIe to communicate between FPGA based processing boards and the host computer and requires transfer of large amounts of data through this channel. But the limited services offered by the FPGA in a standalone slave mode is often a bottleneck, forcing many to use embedded systems on the FPGA, with negative impacts on the performance and available area on the FPGA as consequences. A standalone design however could manage the transfer with higher throughput and smaller footprint on the FPGA if it uses DMA and especially bus master DMA. In this paper, we present a solution to facilitate this task with high efficiency, speed, flexibility and reliability. On the hardware side, our solution uses a bus master DMA design with a 4-lane generation 2 PCIe link and on the software side, using modern Linux kernels, it supports scatter/gather vectored synchronous/asynchronous write, streaming mapping, Message Signaled Interrupt and race condition avoidance mechanism. The software has been verified on several kernels and the hardware has been tested on different FPGAs. The suite has been developed with flexibility and modularity in mind so that it can easily be adapted to different experiment. Currently it is used in the Track Engine Trigger for the IceCube Neutrino Observatory (South Pole) and is also used for command and data transfer in the development of a new high energy physics readout system.