Model-Led Optimisation of a Geometric Multigrid Application

This paper details the construction of an analytical performance model of HYDRA, a production nonlinear multigrid solver used by Rolls-Royce for computational fluid dynamics simulations. The model captures both the computational behaviour of HYDRA´s key subroutines and the behaviour of its proprietary communication library, OPlus, with an absolute error consistently under 16% on up to 384 cores of an Intel X5650-based commodity cluster. We demonstrate how a performance model can be used to highlight performance bottlenecks and unexpected communication behaviours, thereby guiding code optimisation efforts. Informed by model predictions, we implement an optimisation in OPlus that decreases the communication and synchronisation time by up to 3.01 times and consequently improves total application performance by 1.41 times.