Hybrid parallelism in MFIX CFD-DEM using OpenMP

This paper supplements previous work done by Gopalakrishnan and Tafti [21] for the parallel implementation of the Discrete Element Method (DEM) in the open source code Multiphase Flow with Interphase eXchange (MFIX) for distributed memory architectures using MPI. The current study extends the parallel implementation to shared memory architectures using OpenMP making the code amenable to hybrid MPI + OpenMP computations on multicore multiprocessor architectures. The contact force computation, the drag force computation, and locating the particle in fluid cell for neighbor searches are identified to be some of the major time consuming routines and instrumented with OpenMP directives. The OpenMP instrumented code is shown to give a speedup of 7 on 8 threads in a 3D bubbling fluidized bed with 80,000 particles and 18,000 fluid cells. Furthermore, the hybrid performance of MPI + OpenMP is evaluated for a large scale CFD-DEM system of 5.12 million particles. The strong scaling analysis shows good scalability with a speed up of 185 on 256 processors (20,000 particles/processor) for hybrid versus a speed up of 138 for a standalone MPI.