Data format selection for an I/O-intensive large-scale FDTD

One of the bottle-necks of the large-scale FDTD calculation is the data I/O. Although PFS improves the I/O performance it is not always available. Under the computational environment without PVFS and local disks for each PE, limitation on the number of nodes which can produce the output data improves the scalability. HDF5 is compared with the ASCII format and the binary formats from the perspective of the computational efficiency. It becomes clear that HDF5 is beneficial when more than 2⁴ PEs are used and more than 1 GB of data in ASCII format are produced. The effect of the number of files on the datafile production time as seen in Figure 2 and Figure 3 is under investigation for the presentation in the conference. Since it is difficult to install and maintain scientific format libraries, they are usually not present on wide-access HPC systems. This means high portability of the code using any scientific format is currently not achievable.