Session Q49: Extreme-Scale Computational Science Discovery in Fluid Dynamics and Related Disciplines I

From Turbulence Simulations to Petascale Interactive Numerical Laboratories

Scientists in many disciplines would like to compare the results of their experiments or theoretical hypotheses to data emerging from numerical simulations based on first principles. This requires not only that we can run sophisticated simulations and models, but that at least a selected subset of the results of these simulations are available publicly, through an easy-to-use portal. We have to turn our simulations into open numerical laboratories in which anyone can perform their own experiments.  For a scalable analysis we must have an inherently scalable data access. Flat files violate this principle: the user cannot do anything until a very large file has been physically transferred.

The JHU Turbulence databases provide an immersive environment, where users can insert their virtual sensors into the simulation, sending a data stream back to the user. The sensors can be pinned to Eulerian locations or they can move with the flow. They can feed back data on multiple channels, have a variety of operators, e.g.  Laplacian, or various filters. This model also enables users to run time backwards, impossible in a direct numerical simulation involving dissipation. The snapshots are saved frequently enough that one can smoothly interpolate velocities. This simple interface has provided a very flexible, yet powerful way to do science with large data sets from anywhere in the world – we have served over 12 trillion measurements to the community.

Soon we will have Exascale systems, with memory footprints of several petabytes. As a result, only a small fraction of the complete output can ever be saved for later reuse and much of the analysis will have to be done in-situ.  This will make it increasingly harder for scientists outside the core simulation team to reuse this data. The talk will explore ideas how this challenge can be potentially resolved in a satisfactory fashion.   

Towards DNS of Turbulent Combustion at the Exascale

Direct numerical simulation methodology and computing power have progressed to the point where it is feasible to perform DNS in representative of flow configurations encountered in practical combustors. These complex flows encompass effects of mean shear, flow recirculation, and wall boundary layers together with turbulent fluctuations which affect entrainment, mixing, ignition and combustion.  Examples of recent DNS studies with complex flows relevant to gas turbine and internal combustion engines will be presented. These include turbulent premixed combustion with hydrogen/ammonia blends at elevated pressure as a drop-in zero carbon fuel replacement for natural gas in stationary gas turbines and multi-injection autoignition of n-dodecane jets at diesel conditions.  These cases involve complex kinetic interactions with shear flows at elevated temperature and pressure relevant to practical energy applications, and their computational feasibility is dependent on a capable software stack, adaptive mesh refinement and a dynamic task-based programming model tailored for upcoming heterogeneous exascale machines.  Prospects for on-the-fly dimension reduction of complex chemistry to reduce the exorbitant cost of large hydrocarbon chemistry evaluation in DNS will also be described as part of a holistic computational workflow including machine learning enabled by asynchronous task based programming on GPUs.   

Computational Physics on the new Frontier

The Oak Ridge Leadership Computing Facility (OLCF) at Oak Ridge National Laboratory (ORNL) is home to the first exascale supercomputer in the nation, Frontier. Powered by AI-Optimized AMD EPYC CPUs with Radeon Instinct GPUs, Frontier enables extreme-scale simulations in computational physics, including ones involving complex fluid flow, at unprecedented resolutions and performance. In this talk, we share early results of applications with fluid dynamics components and discuss the computational techniques to achieve them on Frontier. We consider how machine learning techniques may be used to accelerate parts of computations, and describe how a leadership computing facility enables access to the massive data sets produced by these simulations.   

Simulation of extreme-scale homogeneous turbulence on a new leadership Exascale GPU platform

Fluid turbulence is a major domain-science problem where high-resolution simulations with trillions of grid points are a powerful research tool. We focus on Exascale-ready GPU algorithm development based on Fourier pseudo-spectral methods suitable for homogeneous turbulence in three-dimensional space. The overall goal is to push the envelope in simulation size while optimizing aggressively for reasonable time to solution. Fast computation on GPUs, efficient data movement between host and device, as well as highly-scalable communication are all crucial for best performance. Use of the latest OpenMP standard promotes inter-platform portability, although machine characteristics still have a strong influence on key programming strategies. Specific issues include the dimensionality of the domain decomposition, collective message-passing via the host or device, efficient host and device memory utilization and opportunities for asynchronism. We shall provide a status report on both the full turbulence simulation code and a 3D FFT kernel of more general interest, on hardware similar to one of the world's first Exascale machines.   

NekRS: A Spectral Element Code for Exascale

We describe NekRS, which is an open source spectral element thermal/fluids code designed for exascale platforms.  NekRS supports both incompressible and low-Mach formulations, ALE-based moving meshes, uRANS, and numerous boundary conditions of relevance to engineering flows.   Second- and third-order timesteppers are provided, using either standard (CFL-limited) semi-implicit methods or characteristics-based subcycling for advection.  A variety of scalable multilevel preconditioners for the pressure Poisson problem have been implemented to ensure optimal performance across a large application space.  Several simulation results are presented, including turbulent flow in the full core of a pebble bed reactor that features 352,000 spherical pebbles in an annular domain.  The mesh features 98.8 million elements of order N=8, for a total of n=50.5 billion gridpoints.  Using 27648 NVIDIA V100s on the OLCF supercomputer, Summit, the wall-clock time for this case is 0.25 seconds per step, corresponding to 6 hours for an entire flow-through time.   

Withdrawn

Reacting flows are pervasive both in our daily lives on Earth and in the Universe. Such flows are fundamental to systems ranging from astrophysical Type Ia supernovae explosions to novel propulsion applications such as scramjets or detonation-based engines. Despite this ubiquity in Nature, turbulent reacting flows still pose a number of fundamental questions concerning their structure and dynamics often exhibiting surprising behavior. In recent years, the advent of massively parallel high-performance computing has enabled the use of large-scale direct numerical simulations (DNS) for the exploration of the reacting flow dynamics in extreme, previously inaccessible regimes. This talk will discuss the computational requirements and challenges associated with such extreme-scale DNS. Furthermore, we will present an overview of a range of phenomena recently discovered in DNS of high-speed reacting flows characterized by high flow speeds, significant compressibility effects, and strong coupling between exothermic reactions and the flow. Such phenomena include intrinsic instabilities of reacting turbulence, onset of catastrophic transitions, e.g., spontaneous detonation formation, and the qualitative changes in the nature of the turbulent cascade in the presence of exothermic reactions.   

Direct numerical simulation of decaying stratified turbulence with extreme scale separation

Stably stratified turbulence (SST) is a model flow for understanding fluid flows that are highly intermittent and anisotropic at large scales.  The understanding derived from SST is important for applications ranging from climate modeling, to pollution mitigation, to deep sea mining, to military operations over cold land or ice. SST is also valuable for enhancing fundamental turbulence theory on turbulent/non-turbulent interfaces, internal intermittency, and anisotropic multi-scale energetics. In all turbulent flows, dynamic range, or the ratio of the largest the smallest length scales, has a profound effect on the fluid dynamics.  Dynamic range is characterized by the Reynolds number, Re.  SST, though, is at least a two parameter problem with Re characterizing the overall dynamic range and Froude number Fr describing the range of comparatively large length scales strongly affected by buoyancy.  In direct numerical simulations (DNSs), at least four decades of scale separation is required to resolve the horizontal scales in the strongly stratified regime.  Here we present results from simulations using up to 24000×24000×6000 grid point and generating about 55 TB per snapshot of the flow field in time.  We show the dynamics when there is sufficient dynamic range for how the flow to be fully turbulent.   

Direct Numerical Simulations of Turbulent, Multiphysics Hypersonic Flows at Extreme Scales

At enthalpies of tens of MJ/kg, the fluid dynamics of high-speed flows past aerodynamically-shaped objects is characterized by shock-turbulence-material interactions.  We seek to understand the details of these interactions through the use of direct numerical simulations of the compressible turbulent reactive flow coupled with material response simulations of the adjacent structure.  Our approach utilizes multiple domain-specific codes built for heterogeneous computing architectures that are coupled through a data exchange and time integration framework.  The presentation will describe these tools through their application on a Mach 6 flow interacting with a 35 degree compression ramp with an embedded compliant panel.  The conditions and model geometry match experiments conducted in the NASA Langley 20-inch Mach 6 tunnel, including a simulated incoming sound field due to tunnel wall boundary layer noise.   

Numerical simulation of the Richtmyer--Meshkov instability evolving from broadband initial perturbations

 This talk presents results from both implicit large eddy simulations (ILES) and direct numerical simulations (DNS), performed using the finite-volume code FLAMENCO, of a mixing layer induced by the Richtmyer-Meshkov instability (RMI) and evolving from amplitude perturbations containing a broad bandwidth of initial modes. In particular, two different broadband perturbations are analysed, defined by their initial radial power spectra P(k)=Ck^m where m=-1,-2. The amplitudes of individual modes are defined such that either (i), the total standard deviation of the two perturbations are the same or (ii), the amplitudes of the highest wavenumber mode are the same. In both cases all modes are initially linear. 

 

The ILES results are used to explore the behaviour of the layer in the high Reynolds number limit at late dimensionless times and demonstrate the persistent influence of initial conditions, as manifested in quantities such as the growth exponent of the mixing layer width θ and the mixing fraction Θ. Specifically, these are shown to depend on the spectral slope m of the initial condition during the period of self-similar growth, which in turn depends on the bandwidth of initial modes. Prior to the development of a turbulent mixing layer, DNS is required for accurate predictions of the transitional behaviour. Direct numerical simulations of the same initial conditions used in the ILES study are performed to illustrate the challenges associated with simultaneously trying to maximise the Reynolds number of the simulation, so that transition through to a fully developed turbulent state may be simulated, as well as trying to maximise the bandwidth of initial modes in order to be representative of the initial surface perturbations found in real flows. Some preliminary results from the simulations will be presented and possible extensions of the current work  in order to achieve higher Reynolds numbers and bandwidths will also be discussed.   

Geometry dynamics of turbulent flow structures via tracking

We present a methodology for the analysis of the temporal evolution of flow structures and their mutual interactions based on tracking. The structures, defined as closed surfaces from instantaneous snapshots of the flow, are characterized by area-based joint PDFs of differential geometry pointwise quantities (curvedness and shape index). A hybrid region- and attribute-based tracking algorithm finds realizable correspondences between structures at consecutive time frames, constructing a directed graph that describes the structure evolution and interaction events (e.g., merging and splitting). Common patterns of structure evolution are extracted by graph querying and subgraph mining conditioned on geometric attributes. Ensemble statistics for structures with common geometries are obtained to study their dynamics through trajectories in parameter spaces that combine geometry and flow physics. We show application of the methodology to numerical datasets obtained from direct numerical simulation of turbulent flows in several canonical configurations, including decaying homogeneous isotropic compressible turbulence and shock-turbulence interaction (at different Reynolds, shock and turbulence Mach numbers), compressible mixing layers (at different convective Mach numbers), and the multiphase flow interaction of a droplet breakup in homogeneous isotropic incompressible turbulence.   

The Challenges of Modeling Astrophysical Reactive Flows

Energy production in stellar environments is dominated by thermonuclear energy release, which can take place in quiet convective flows or explosive environments.  Multidimensional models of astrophysical reactive flows have many challenges: capturing the relevant length and timescales, and keeping the different physics inputs coupled together over the course of the simulation.  We will show applications to X-ray bursts, thermonuclear supernovae, and massive star evolution and discuss how algorithmic improvements and harnessing the power of GPUs has allowed us to make significant progress on understanding these astrophysical events, using our open source AMReX-Astrophysics suite of simulation codes.   

Asynchronous exascale PDE solvers: exploiting extreme parallelism for turbulence simulations

Future exascale computing systems will be available to study

multi-physics multi-scale natural phenomena and engineering systems.

Many of these can be described by partial differential equations (PDEs)

that may be tightly coupled with long-range correlations. A prime example

is high-Reynolds number turbulence

which is critical in phenomena as diverse as mixing in an engine, pollutants in

the atmosphere, aerodynamic drag, and the structure of the observable universe.

Because of the exceedingly complex nature of the governing equations,

little is known from its analytical treatment and most advances

have relied on numerical simulations.

The relentless increase in computational power

has enabled simulations at more realistic conditions opening the door

to important scientific breakthroughs.

This explosion in computational power has been realized through massive

parallelism. Current simulation approaches present significant challenges

with the extreme levels of parallelisms expected on exascale systems,

In this talk we first discuss these computational

challenges in current approaches and introduce a novel concept

for simulations which exploits relaxed synchronizations between processing elements.

We show how this approach can effectively mitigate

(or eliminate) synchronization and communication overheads

which are well-known bottlenecks expected at exascale.

We will show how new errors emerge because of asynchrony and how

new numerical schemes can be designed to retain high accuracy.

These schemes are found to be able to accurately capture turbulence

in realistic simulations and extend the scalability of codes

significantly. We will show results from simple model problems,

to compressible turbulence and detonations.

We will conclude with a perspective on the new opportunities

presented by asynchronous schemes at exascale and future work needed.   

Fluid Dynamics of Supernova Remnants

Supernovae – explosions of stars – are a central problem in astrophysics since they encapsulate the entire process of stellar evolution and nucleosynthesis. Rayleigh-Taylor (RT) and Richtmyer-Meshkov (RM) instabilities, developing during the supernova blast, lead to intense mixing of the star’s materials and couple astrophysical to atomic scales. We handle fluid dynamics challenges of RT/RM problem by directly linking the conservation laws governing RT/RM dynamics to symmetry-based momentum model, by precisely deriving the model parameters in the scale-dependent and scale-invariant regimes, and exactly integrating the model equations for variable acceleration in the scale-dependent linear and nonlinear regimes and in self-similar mixing regime. The theory outcomes explain the observations of supernova remnants, yield the design of scaled laboratory experiments for quantification of RT/RM dynamics in high energy density settings, and find that supernovae can indeed be regarded as an astrophysical initial value problem.