Direct numerical simulations of turbulent flows using high-order asynchrony-tolerant schemes: Accuracy and performance

Direct numerical simulations (DNS) are an indispensable tool for understanding the fundamental physics of turbulent flows. Because of their steep increase in computational cost with Reynolds number (R-lambda), well-resolved DNS are realizable only on massively parallel supercomputers, even at moderate R-lambda. However, at extreme scales, the communications and synchronizations between processing elements (PEs) involved in current approaches become exceedingly expensive and are expected to be a major bottleneck to scalability. In order to overcome this challenge, we developed algorithms using the so-called AsynchronyTolerant (AT) schemes that relax communication and synchronization constraints at a mathematical level, to perform DNS of decaying and solenoidally forced compressible turbulence. Asynchrony is introduced using two approaches, one that avoids synchronizations and the other that avoids communications. These result in periodic and random delays, respectively, at PE boundaries. We show that both asynchronous algorithms accurately resolve the large-scale and small-scale motions of turbulence, including instantaneous and intermittent fields. We also show that in asynchronous simulations the communication time is a relatively smaller fraction of the total computation time, especially at large processor count, compared to standard synchronous simulations. As a consequence, we observe improved parallel scalability up to 262144 processors for both asynchronous algorithms. (C) 2020 Elsevier Inc. All rights reserved.