The PreSync Project: Synchronization Automation in the Cactus Framework

The Einstein Toolkit (ETK) is an open-source software platform for computational simulations in relativistic astrophysics and gravitational physics. The Cactus Framework (Cactus) comprises the core component of the ETK and handles creation of distributed data structures, parallelism, I/O, checkpointing, etc. The current Cactus scheduling system relies on the manual synchronization of ghost zones for grid functions (i.e. distributed matrices). Unfortunately, deciding which subroutines should synchronize which grid functions is a non-trivial problem. Incorrect synchronization can result in over-synchronization (a performance problem) or under-synchronization (an error). This issue also creates a barrier for new users and collaborators. We have developed a new scheduling and synchronization method for Cactus to improve the accessibility and efficiency of the ETK. Synchronization and application of boundary conditions are now handled automatically by Cactus, and performance has been improved by as much as twenty percent. The new approach requires each subroutine to have "read" and "write" declarations for individual grid functions. Before scheduled subroutines run, Cactus checks these declarations and performs any synchronization or boundary condition updates as needed. This method removes the difficulty of deciding where synchronization should take place and removes unneeded synchronizations. The only knowledge required to provide these declarations is what grid functions a given subroutine uses and on which parts of the grid they are read or written. In addition, these changes have reduced the number of synchronizations by approximately five percent in unigrid simulations. This new system is a first step in improving Cactus' scheduling methods to use more advanced and scalable techniques.