Optimizing finite element programs on the Cray X1 using coloring schemes

Using the Environmental Quality Modeling program FEMWATER as a test-bed code, 27 percent of the time needed to run a given groundwater flow application on the ERDC Cray X1 using four multistream processors (MSPs) was spent assembling the global stiffness matrix. This poor performance is because the above code cannot multistream without help. The technique of "coloring" the elements makes it possible to multistream this section of the code, thus taking advantage of the hardware capability of the machine. Coloring for assembling the global stiffness matrix involves dividing the elements into different groups such that no node point touches any elements with the same color. This paper presents a simple coloring algorithm in FORTRAN and show how it was implemented into FEMWATER to achieve multistreaming on the ERDC Cray X1. It then gives a detailed description on how the program was modified, what compiler options were used, and what compiler directives worked best. Finally, timing results are given. Some programs that have good MPI (or equivalent) communication are better suited for running in the single-streamed processor (SSP) mode. In the SSP mode, coloring of the elements is not needed for assembling the global stiffness matrix. Timings for running in the SSP mode are shown, too.