H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing with 3D Adaptive Mesh Refinement and Local Adaptive Time Stepping

General relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multifaceted optimizations that, collectively, accelerate computation by 2-5 orders of magnitude for a wide range of applications. First, it introduces a spherical grid with 3D adaptive mesh refinement that operates in each of the three dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-resolution computational performance. Second, we demonstrate that local adaptive time stepping on a logarithmic spherical-polar grid accelerates computation by a factor of less than or similar to 10 compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of similar to 10(9) zone cycles per second per node on 5400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We illustrate H-AMR's computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of 13,440 x 4608 x 8092 cells and a total of less than or similar to 22 billion cells and similar to 0.65 x 10(8) time steps, it is among the largest astrophysical simulations ever performed. We find that frame dragging by the black hole tears up the disk into two independently precessing subdisks. The innermost subdisk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields.

Automated summary

General relativistic magnetohydrodynamic simulations have provided revolutionary insights into black hole accretion. The GPU-accelerated GRMHD code H-AMR presented in this study demonstrates significant computational speed improvements, allowing for large-scale astrophysical simulations. The simulation performed using H-AMR reveals the tearing of a tilted thin accretion disk by a rapidly spinning black hole and provides evidence of alignment between the disk and black hole spin.