Multilevel parallelization models: application to VIV

Realistic simulations of flow past a flexible cylinder subject to vortex-induced vibrations require a large number of Fourier modes along the cylinder span and high resolutions in the streamwise and cross-flow directions. Parallel computations employing a single-level parallelism for this type of problems have clear performance limitations that prevent effective scaling to the large processor count on modern supercomputers. We present two multilevel parallel paradigms based on MPI/MPI and MPI/OpenMP for high-order CFD methods within the spectral element framework and compare their performance. In the MPI/MPI model, we employ MPI process groups/communicators to decompose the flow domain and MPI processes into different levels. In the MPI/OpenMP model, we employ multiple OpenMP threads to split the workload within the subdomain and take a coarse-grain approach that significantly reduces the OpenMP synchronizations. For identical configurations the MPI/MPI model is observed to be generally more efficient. However, for dynamic p-refinement the MPI/OpenMP approach is more effective. Because a greatly reduced number of processes are involved in the communications at each level, these multilevel parallel paradigms reduce the network latency overhead and enable the applications to scale to a large number of processors more efficiently.