P-refinement and P-threads Original Research Article

P-type refinement leads to exponential decay of numerical errors for sufficiently smooth solutions and has been used effectively in turbulence and structural mechanics simulations in the context of spectral and hp finite element discretizations. However, it induces a computational cost of O(Pd+1) in d dimensions, which is higher than lower-order methods. In this paper, we demonstrate that by employing multi-threading within MPI processes we manage to counter-balance the cost increase associated with P-refinement. This approach reduces effectively the wall clock time, and keeps it essentially constant as the polynomial order is increased while achieving exponential convergence rate. Since the number of threads within MPI processes can be dynamically adjusted through thread library functions, the algorithm can be readily adapted for dynamic P-refinement. The resulting hybrid MPI/threads dual-level parallelism is particularly suitable for modern supercomputers consisting of “symmetric multiprocessor” nodes. We demonstrate this approach in simulations of two three-dimensional fluid dynamics problems.