Parthenon - A Performance Portable Block-Structured Adaptive Mesh Refinement Framework

On the path to exascale the landscape of computer device architectures and corresponding programming models has become much more diverse. While various low-level performance portable programming models are available, support at the application level lacks behind. To address this issue, we present Parthenon, a performance portable block-structured adaptive mesh refinement (AMR) framework, derived from the well-tested and widely used Athena++ astrophysical magnetohydrodynamics code but generalized to serve as the foundation for a variety of downstream multi-physics codes. Parthenon adopts the Kokkos programming model and provides various levels of abstractions including multi-dimensional variables, packages defining multi-physics components, and compute kernels on device architectures. Parthenon allocates all data in device memory to reduce data movement, supports the logical packing of variables and mesh blocks to reduce the number of kernels and thus mitigate kernel launch overhead, and employs one-sided, asynchronous MPI calls to reduce communication overhead in multi-node simulations. Using a hydrodynamics miniapp, we demonstrate weak and strong scaling on various architectures including AMD and NVIDIA GPUs, Intel and AMD x86 CPUs, IBM Power9 CPUs, as well as Fujitsu A64FX CPUs. At the largest scale on Frontier (the first TOP500 exascale machine), the miniapp reaches a total of 1.7×10¹³ zone-cycles/s on 9,216 nodes (73,728 logical GPUs) at ~92% weak scaling parallel efficiency (starting from a single node). In combination with being an open, collaborative project, this makes Parthenon an ideal framework to target exascale simulations in which the downstream developers can focus on their specific application rather than on the complexity of handling massively-parallel, device-accelerated AMR. Finally, we present existing open source astrophysical downstream codes, including AthenaPK (MHD), Phoebus (GRMHD), and KHARMA (GRMHD), that already leverage Parthenon.