Session K60: Extreme Scale Computational Science Discovery in Fluid Dynamics and Related Disciplines II

Invited Talk: Kenneth Jansen

For several decades, scale-resolving turbulence simulations have served as grand challenge problems capable of saturating the largest available computer’s resources for as long (or longer) than the resource was available.  Larger machines have allowed simulations at larger Reynolds numbers but that growth is slow due to the three-dimensional and unsteady nature of turbulence that drives the cost (degree-of-freedom count multiplied by time step count) to scale with the Reynolds number to powers  ranging from roughly 1 to 4 depending upon the scale-resolving method chosen with higher exponents coming from more scales resolved vs. modeled. 

While exponents on the Reynolds numbers are important, larger machines have also allowed expansion of geometric and flow complexity.  In this talk, we will discuss the development of finite element methods for application to scale-resolving simulations on extreme scale computational resources.  The methods developed provide very low dissipation and higher order accurate discretizations. Their ability to use unstructured grids to match grid resolution to the local needs of the scale resolving simulation makes them particularly attractive and efficient for complex geometry flows and more fundamental flows with a large spatial variation in the smallest required scale that must be resolved.  The talk also provides a contrast of classical and newer approaches to equation formation, assembly, and solution that accounts for emerging hardware.

It is important to note that getting the solver to scale well and make efficient use of the emerging hardware is only one of the challenges of extreme scale computing. Other significant challenges addressed in this talk include preparing the inputs for simulations at this scale (pre-processing) and extracting meaningful insight from the massive spatial-temporal data stream that a peta and soon exascale turbulence simulation produces.   Insight extraction in the classical sense, writing data to files, becomes challenging due the relatively slow growth of I/O resources relative to computational resources which motivates development of in situ data analytics and compression techniques.  This talk will discuss the general and specific challenges that have been addressed for unstructured grids in this area and will close with demonstrations.   

A machine learning approach for second moment closure modeling of stably stratified turbulence

We use machine learning (ML) for closure modeling of the Reynolds Averaged Navier Stokes (RANS) equations applied to stably stratified turbulence (SST). SST is strongly affected by fine balances between forces and, in decaying cases, becomes more anisotropic in time. Moreover, there is a limited understanding of the physical phenomena described by some of the terms in the RANS equations. Rather than attempting to model each term separately, it is attractive to see the capability of machine learning to model groups of terms, i.e., to directly model the force balances. We consider decaying SST that is homogeneous and stably stratified by a uniform density gradient, enabling dimensionality reduction. Training data is from Massive Scale Direct Numerical Simulations (MSDNS) with up to 3 trillion grid points in order to span the largest range of buoyancy Reynolds number (Gn) and Froude number (Fr) possible on existing computers. The effectiveness of ML to model flow evolution parameterized in the Gn-Fr space is shown. Furthermore, we demonstrate the capability of the latest high-performance computing architectures utilizing GPUs to distributedly deploy the ML models.   

An Optimized Species-Conserving Monte Carlo Method with Potential Applicability to High Entropy Alloys

 We present a species-conserving Monte Carlo (MC) method, motivated by systems such as high-entropy

alloys. Current fast local-structure MC methods do not conserve the net concentration of atomic species, or

are inefficient for complex atomic systems. By coarse-graining the atomic lattice into clusters and developing

a renormalized MC method that takes advantage of the local structure of the atoms, we are able to significantly

reduce the number of iterations required for MC simulations to reach equilibrium. In addition, the structure of

the method enables easy parallelizability for the future.   

Tensor network methods for solving the Vlasov equation

It has been proposed that tensor networks can be used to approximately solve partial differential equations with exponential speed-up, provided that the PDE is compressible and can be efficiently represented within the tensor network framework. Previously, we presented results on 2-D simulations of the Vlasov equation, which provides an ab-initio description of high-temperature plasmas. Here, we consider simulations in up to 6-D and analyze the performance of the tensor network methods in these higher dimensional calculations. For the test cases studied, we find that we are able to capture important features of the dynamics even with significant amounts of compression.   

Tensor networks for many-body quantum transport

Tensor networks are making rapid progress in the efficient simulation of open quantum systems with non-Markovian interactions with particle reservoirs or heat baths. One such scenario is quantum transport through impurities, where external reservoirs-the environment-drive particles through a junction region, which requires both a machinery to handle many-body interactions (tensor networks), a representation of the reservoirs, and a computational paradigm that properly embodies the physical scenario. Exploiting all three pillars enables simulating many-body transport in the presence of an extensive, non-Markovian electronic reservoirs [1,2].

[1] Marek M. Rams and Michael Zwolak, Phys. Rev. Lett. 124, 137701 (2020)

[2] Gabriela Wójtowicz, Justin E. Elenewski, Marek M. Rams, and Michael Zwolak, Phys. Rev. A 101, 050301(R) (2020)   

Disentangling Interacting Systems with Fermionic Gaussian Circuits: Application to the Single Impurity Anderson Model

Tensor network representations of quantum many-body states provide a powerful tool to describe strongly correlated systems. By applying a specific change of basis, defined by a quantum circuit obtained when compressing a fermionic Gaussian state, we find drastically improved computational efficiency. As an example of the approach, we study the 1D single impurity Anderson model and find the basis transformation helps in interpreting the impurity physics. The Gaussian multi-scale entanglement renormalization ansatz circuits are introduced, where emergent coarse-grained models are studied both in terms of entanglement and time evolution.   

Performance of real space Quantum Monte Carlo and QMCPACK on GPU accelerated Exascale supercomputers

We report the performance of real space quantum Monte Carlo calculations performed with QMCPACK for materials on NVIDIA and AMD GPUs, as found in latest generation of supercomputers. Over the last few years QMCPACK (https://www.qmcpack.org) developers have developed a new performance portable design that enables efficient support for both CPU and GPU accelerated systems from a largely common set of code paths. The design supports full fallback to CPU execution where features have not been ported to or optimized for GPUs. Here we report the performance obtained for a wide range of system sizes and discuss approaches to extend the complexity of materials that can be efficiently studied within current memory limits. The experiences are expected to be broadly relevant to other electronic structure codes and algorithms.

 

    

Accumulative reservoir construction: Bridging continuously relaxed and periodically refreshed extended reservoirs

The simulation of open many--body quantum systems is challenging, requiring methods to both handle exponentially large Hilbert spaces and represent the influence of (infinite) particle and energy reservoirs. These two requirements are at odds with each other: Larger collections of modes can increase the fidelity of the reservoir representation but come at a substantial computational cost when included in numerical many--body techniques. An increasingly utilized and natural approach to control the growth of the reservoir is to cast a finite set of reservoir modes themselves as an open quantum system. There are, though, many routes to do so. Here, we introduce an accumulative reservoir construction---an ARC---that employs a series of partial refreshes of the extended reservoirs. Through this series, the representation accumulates the character of an infinite reservoir. This provides a unified framework for both continuous (Lindblad) relaxation and a recently introduced periodically refresh approach (i.e., discrete resets of the reservoir modes to equilibrium). In the context of quantum transport, we show that the phase space for physical behavior separates into discrete and continuous relaxation regimes with the boundary between them set by natural, physical timescales. Both of these regimes ``turnover'' into regions of over-- and under--damped coherence in a way reminiscent of Kramers' crossover. We examine how the range of behavior impacts errors and the computational cost, including within tensor networks. These results provide the first comparison of distinct extended reservoir approaches, showing that they have different scaling of error versus cost (with a bridging ARC regime decaying fastest). Exploiting the enhanced scaling, though, will be challenging, as it comes with a substantial increase in (operator space) entanglement entropy.   

A highly efficient delayed update algorithm for evaluating Slater determinants in quantum Monte Carlo

As cutting edge quantum Monte Carlo simulations demand thousands of electrons in a simulation cell, matrix operations related to Slater determinants lead the computational cost. McDaniel et at. [1] proposed a delayed update algorithm to maximize computational efficiency by leveraging matrix-matrix multiplication when updating the inverse matrices of Slater determinants. However, preparing intermediate matrices for applying the Sherman-Morrison-Woodbury formula remains a bottleneck. In this work, we eliminate that bottleneck by iteratively updating those intermediate matrices and also show the full scheme of integrating the delayed update algorithm into a step of single electron move. In addition, we also extend our algorithm to accommodate the compute characteristics of GPUs. We will demonstrate the high efficiency of our algorithm, as implemented in QMCPACK, on CPUs and GPUs

[1] T. McDaniel, E. F. D’Azevedo,Y. W. Li, K. Wong, and P. R. C. Kent, The Journal of Chemical Physics 147, 174107 (2017)   

Towards Exascale Hybrid Electronic-Structure Theory Calculations beyond 10,000 Atoms

The advent of exascale computing paves the way for more accurate first-principles predictions that can target complex materials under realistic conditions. However, this requires electronic-structure theory codes that are specifically optimized to harvest the computational power of massively parallel CPU/GPU architectures. This challenge is addressed in the EU Centre of Excellence NOMAD by developing code-independent libraries for the computationally dominant operations. In this work, we present recent algorithmic advancements in memory parallelization and load-distribution that improve the performance of hybrid-functional calculations by two orders of magnitude compared to the original implementation in FHI-aims [1]. This linear-scaling evaluation of exact exchange integrals enables investigating systems with several 10,000 atoms, in the limit of which the cubically scaling eigenvalue solvers become computationally dominant. In this regard, we report recent optimizations of the ELPA library [2], including their adaption to several different GPU architectures, that help in mitigating this hurdle.

[1] S. V. Levchenko et al., Comp. Phys. Comm. 192, 60 (2015). 

[2] P. Kus et al., Par. Comp. 85, 167 (2019).   

Machine learning assisted reverse Monte Carlo modeling for neutron total scattering data

In the area of atomic-level structure modeling, there are two well known parallel problems. The theory driven modeling usually cannot fully account for the disorder of practical system and therefore may fail to reproduce the complete picture of structure model as observed experimentally. The data driven approach tries to derive the strucrual model from the experimental data in a reverse manner (i.e., data to model) and therefore natually is able to catch features observed experimentally. But quite often it lacks the accurate coverage of energetic landscape from the theoretical perspective. In this contribution, we aim at bringing in a novel approach combining the theoretical and experimental considerations. To realize this, the LAMMPS module for energy calculation is implemented into the reverse Monte Carlo routine (here, the RMCProfile package was used) for modeling total scattering data. Through such a combined approach, atomic positions would be adjusted according to the agreement with experimental scattering data and the energy landscape simultanesouly. Specifically concerning the energy calculation, the Gaussian processing based machine learning routine for potential field construction is employed here. Such an approach, at the same time providing density functional theory level of accurary, gurantees a reasonably short computational time which is required for the metropolis algorithm for structure modeling. The LAMMPS implemented RMCProfile package for conducting the combined modeling is generally applicable to utilize neutron and X-ray total scattering data, X-ray absorption spectroscopy data, elecrton scattering data, etc. for structure modeling to provide insights into structure-property link of general consensed matter systems.   

Numerical modeling of dielectric barrier discharge actuators based on the properties of low-frequency plasmons

Electrohydrodynamic flow control systems have proven to be among the most promising flow control strategies within previous decades. Several methods are available for efficient evaluation and description of such systems' effects. Yet, due to these systems' critical role in various applications, possible improvements are still being investigated. A new phenomenological model is presented for the simulation of the plasma actuators based on the electrodynamic properties of low-frequency plasmons. The model simulates the plasmonic region as a dispersive medium. This dissipated energy is added to the flow by introducing a high-pressure region, calculated in terms of local body force vectors, requiring the distribution of the electric field and the polarization field. The model determines the electric field for the computation of the body force vector based on the Poisson equation and implements the simplified Lorentz model for the polarization field. To fully explore the performance of the presented model, an experiment has been conducted, comparing the observed effect of plasma actuators on the fluid flow with the results predicted by the model. The model is then validated based on the results of other distinct experiments and exempted numerical models based on the exchanging momentum with the ambient neutrally charged fluid, demonstrating that the model has improved adaptability and self-adjusting capability compared to the available models.   

Efficient compression of classical functions using tensor networks

A classical function discretized on 2ⁿ points can be embedded in the coefficients of an n-qubit state.  If this state has low entanglement, it can be efficiently represented as a tensor network, and in particular as a matrix product state (MPS) when the classical function is low-dimensional.  This approach has been demonstrated to give a substantial speedup in solving differential equations appearing in fluid dynamics and plasma physics, among other areas.  In this talk, I will show exact low-bond-dimension MPS representations of important classes of functions including Fourier series and polynomials.  I will then argue more generally what types of functions can be represented efficiently, and hence in which physical contexts an MPS-based function compression could be useful.