Effects of multithreading on data and workload distribution for distributed-memory multiprocessors

While data and workload distribution can be tailored to fit a particular problem to a particular distributed-memory architecture, it is often difficult to do so for various practical issues. This paper presents our study on multithreading for distributed-memory multiprocessors. Specifically, we investigate the effects of multithreading on data distribution and workload distribution with variable thread granularity. Various types of workload distribution strategies are defined along thread granularity. Three types of data distribution strategies are investigated: row-wise cyclic, k-way partial-row cyclic and blocked distribution. We have implemented all of these on the 80-processor EM-4 distributed-memory multiprocessor using highly-sequential Gaussian elimination with partial pivoting and highly-parallel matrix multiplication. Experimental results indicated that multithreading can offset the loss that is due to the mismatch of data distribution to workload distribution for even sequential and irregular problems while giving high absolute performance