Extending Work Queue of HCA Endpoint Using Software Work Queue Approach: Experimental Evaluation with uDAPL

With an ever increasing demand for computing power, number of nodes to be deployed in a high performance cluster is increasing. This has put stress on various hardware resources like endpoints (equivalent to Queue Pairs), memory etc. One of the factors limiting number of endpoints with a Host Channel Adaptor (HCA) is the amount of total descriptor space available in the form of on-board memory. For the same amount of descriptor space available, number of endpoints supported by an HCA is inversely proportional to the depth of work queue (WQ) for an endpoint (EP). Therefore to support more number of endpoints, depth of WQ has to be decreased. In this paper, we present an optimal approach of extending WQ using Software Work Queue (SWQ). SWQ is an extension to the work queue of EP on HCA. It provides a larger pool of descriptors (i.e. work requests) per EP to an application (e.g. uDAPL application, MPI library) than that available in the hardware. The advantage of this technique lies in the fact that more number of endpoints are supported by HCA without costing the depth of the WQ. Experimental evaluation suggests positive impact as requests in SWQ are given to hardware work queue (HWQ) as soon as it gets depleted.