Effective dynamic load balance using space-filling curves for large-scale SPH simulations on GPU-rich supercomputers

Billion of particles are required to describe fluid dynamics by using smoothed particle hydrodynamics (SPH), which computes short-range interactions among particles. In this study, we develop a novel code of large-scale SPH simulations on a multi-GPU platform by using the domain decomposition technique. The computational load of each decomposed domain is dynamically balanced by applying domain re-decomposition, which maintains the same number of particles in each decomposed domain. The performance scalability of the SPH simulation is examined on the GPUs of a TSUBAME 2.5 supercomputer by using two different techniques of dynamic load balance: the slice-grid method and the hierarchical domain decomposition method using the space-filling curve. The weak and strong scalabilities of a test case using 111 million particles are measured with 512 GPUs. In comparison with the slice-grid method, the performance keeps improving in proportion to the number of GPUs in the case of the space-filling curve. The Hilbert curve and the Peano curve show better performance scalabilities than the Morton curve in proportion to the increase in the number of GPUs.