The dynamic nature of Congestion inInfiniBand

The InfiniBand Congestion Control (CC) mechanism is able to reduce congestion and improve performance in many situations. In this paper we study the characteristics of congestion in InfiniBand by monitoring and analyzing the CC mechanism with a hardware analyzer. To the best of our knowledge, this is the first paper that presents experience with, and analysis of, the InfiniBand CC with such a tool. We found that there can be more than one “root of congestion”, as defined by the IBTA specification, existing at the same time in the congestion tree, and a “root of congestion” can be converted to a “victim of congestion” as its nature changes. We also observed that even with constant traffic flows, the “root of congestion” will shift from one place to another within the congestion tree, with corresponding consequence for packets from various traffic sources: traffic that might be negatively impacted by tree spreading and might be not contributing to the “root of congestion” before, will be treated as a congestion contributor and then be throttled by the CC mechanism.