Session D40: Building the Bridge to Exascale: Applications and Opportunities for Materials, Chemistry, and Biology III

Preparing for exascale: additive manufacturing process modeling at the fidelity of the microstructure

In FY17, the USDOE Exascale Computing Project (ECP) initiated projects to design and develop simulation codes to use exascale computing. This application development is organized around computational motifs. Here, we present an overview of the motifs of computational materials science, from the “particles” using by molecular dynamics to the “grids” using by phase-field models and the various solution algorithms such as FFTs. Examples will be taken from the co-design centers ExMatEx and CoPA, as well as the application development project ExaAM. This project includes an integration of all the computational components of the metal additive manufacturing (AM) process into a coupled exascale modeling environment, where each simulation component itself is an exascale simulation. What has emerged is that exascale computing will enable AM process modeling at the fidelity of the microstructure. Here we discuss what this means, in particular, tight coupling of Process-Structure-Property calculations. Macroscopic continuum codes (ALE3D, Truchas and OpenFOAM) are used to simulate melt-refreeze, within which mesoscopic codes (Phase-field and Cellular Automata) are used to simulate the development of material microstructure. This microstructure is then used by polycrystal plasticity codes (ExaConstit) to calculate local material properties. The project is driven by a series of demonstration problems that are amenable to experimental observation and validation. We present our coupled exascale simulation environment for additive manufacturing and its initial application to AM builds.   

Enabling First Principles Multiscale-Multiphysics Simulations of Complex Thermo-Fluid Systems Through Exascale Computing

Understanding and controlling turbulence, aerothermodynamics, and propulsion processes in advanced thermo-fluid systems presents many challenges. A multitude of strongly coupled fluid dynamic, thermodynamic, transport, chemical, and heat transfer processes are intrinsically coupled and must be considered simultaneously in complex domains. These multiscale physics are not currently understood or modeled with sufficient accuracy. Without their inclusion, timely Research and Development of advanced systems will be significantly deficient. Exascale computing offers significant opportunities treat these physics with unprecedented accuracy and speed. However, the foundational hybrid-CPU+GPU architectures present many challenges to exploit their full potential power. This presentation will highlight the inherent challenges associated with porting complex multiphysics solvers to these architectures and the approach taken to achieve optimal performance using the RAPTOR code framework developed by Oefelein et al. as an example application.   

Generating a Comprehensive Map of Cancer Morphology in Whole Slide Tissue Specimens

Advanced imaging technologies can capture extremely high-resolution images of tissue specimens, and quantitative analyses of cancer morphology using these images have shown value in a variety of correlative and prognostic studies. Our work on Summit will generate a comprehensive multi-scale mapping of cancer morphology with a dataset of more than 10,000 whole slide tissue images from over 20 cancer types. The work will use a collection of deep learning analysis pipelines we have developed to study, quantify and characterize tissue structure in diseased and normal tissue specimens. These analysis pipelines generate distributions of nuclei and cells and patch-level maps of lymphocyte distributions and segmentations of tumor regions. The analysis results will provide a first-ever representations of lymphocyte maps, nuclear characterizations and characterizations of tumor regions on a dataset of this scale. We expect that studies supported by these rich datasets will enable the development of biomarkers to predict clinical outcome and a better epidemiological understanding of cancer subtypes and how constituent cells contribute to cancer invasion and expansion.   

Operator Dynamics in Quantum Circuits with Subsystem Symmetry

Our understanding of quantum dynamics in many-body quantum systems has been revolutionized in recent years by the study of random quantum circuits. These models provide a tractable setting in which we can understand ideas such as thermalization, operator spreading and quantum chaos. Furthermore, it is known that a richer variety of phenomena can occur when such models are enriched with a set of symmetries. We will focus on a particularly exotic set of symmetries which act on lower dimensional sub-manifolds of our system, a situation which is relevant to the study of highly quantum 'fracton' phases of matter. I will discuss approaches we may take to simulate such quantum dynamics numerically in higher dimensional systems. In particular, we will see that a restricted class of automaton circuit dynamics can be efficiently simulated while retaining the essential attributes of generic quantum chaotic systems. This technique will allow us to understand the properties of circuits with subsystem symmetry and may provide a valuable new tool for future studies of chaotic quantum dynamics.   

Radiation-matter interaction in graphene molecules: implementation on Geant 4 and computational simulations

The voxelized computational approximation of the graphene molecules for different amounts of carbon atoms was simulated, using computational programs with optimized Geant 4 type geometry. The computational simulation of radiation transport throught matter with characteristic interactions in the Uv-Vis spectral range, were made for radiation sources, detectors and graphene molecules. The optimization of the theoretical Uv-Vis spectra obtained from Geant 4, was achieved through algorithms development on Matlab. A method for computational reconstruction of UV-Vis spectra was proposed. The results suggest that is possible observe the contributions of the conjugated pi and sigma bonds, in the UV-Vis characteristic spectra as expected. Also, it was found that the variation in the size of the graphene molecules, influence the height of the band associated with the sigma bond, in agreement with experiments broadly studied. As well as, the proposed methodology suggests Geant 4 as a potential tool to simulate radiation-matter interactions in graphene-based molecules.   

ExaTN - A Scalable Exascale Math Library for Hierarchical Tensor Network Representations and Simulations in Quantum Many-Body Theory and Beyond

Tensor network theory has recently paved the path to efficient numerical simulations of two- and three-dimensional many-body Hamiltonians describing strongly correlated quantum particles, but it still requires efficient software infrastructure that scales well on leadership heterogeneous HPC systems. To address this need, we develop ExaTN: A scalable math library for processing hierarchical tensor representations. Our library enables the use of advanced hierarchical tensor network states capable of expressing local expectation values in strongly entangled quantum systems efficiently. ExaTN allows building arbitrarily complex tensor networks for which it exposes a set of high-level API functions which automate tensor optimization procedures. A highly modular design of ExaTN allows seamless switching of computational backends for the computer system of choice, from a laptop to a leadership GPU-accelerated HPC platform, like Summit. The internal task-based parallel runtime then assures a load-balanced execution of tensor processing workloads.   

Porting ITensor to Julia

In this talk, we present ITensors.jl, a ground-up rewrite of the C++ ITensor library in Julia. ITensor is a leading software package for simulating quantum many-body systems with tensor networks. Julia is a relatively young just-in-time (JIT) compiled language that is particularly well suited for scientific computing. We will discuss the advantages and disadvantages of moving from C++ to Julia, including ease of development and performance. We will also discuss new designs for the Julia version that are in development or planned, such as a rewrite of the sparse tensor library optimized with multithreading, new tensor contraction backends, automatic fermion sign support, GPU support, and automatic differentiation for the automated optimization of tensor networks.   

Breaking the entanglement barrier: Tensor network simulation of quantum transport

The recognition that large classes of quantum many-body systems have limited - or efficiently representable - entanglement in the ground and low-lying excited states led to dramatic advances in their numerical simulation via so-called tensor networks. However, global dynamics elevates many particles into excited states, and can lead to macroscopic entanglement (seen both experimentally and theoretically) and the failure of tensor networks. Here, we show that for quantum transport - one of the most important cases of this failure - the fundamental issue is the canonical basis in which the scenario is cast: When particles flow through an interface, they scatter, generating a "bit" of entanglement between spatial regions with each event. The frequency basis naturally captures that - in the long time limit and in the absence of an inelastic event - particles tend to flow from a state with one frequency to a state of identical frequency. Recognizing this natural structure yields a striking - exponential in some cases - increase in simulation efficiency, greatly extending the attainable spatial and time scales. The concepts here broaden the scope of tensor network simulation into hitherto inaccessible classes of non-equilibrium many-body problems [see arXiv:1904.12793].   

Accelerate Science on Perlmutter with NERSC

Towards exascale computing, the National Energy Research Scientific Computing (NERSC) Center has procured a ~100 PetaFLOP/s supercomputer called Perlmutter. This talk will give an overview of its architectural details and discuss what Perlmutter can offer to the scientific community especially to Material Science and Chemistry. These offerings not only include cutting-edge hardware and technology but also highly optimized software stack and expert user support. The NERSC Exascale Science Application Program (NESAP) provides resources such as hackathons with performance engineers, early access to hardware, and NERSC-funded PostDocs to select application teams, and lessons learned from these teams are then disseminated to the general community. NERSC also collaborates with vendors and other High Performance Computing (HPC) developers on math libraries, performance models and tools, compiler development and performance portability. With an emphasis on Material Science and Chemistry, we will pinpoint the opportunities that Perlmutter and NERSC can bring for exascale and beyond.   

Central Moment Lattice Boltzmann Method with Fokker-Planck Guided Collision for Non-Equilibrium Flows

Central moments-based lattice Boltzmann method (LBM), a recent approach for flow simulations, is generally based on the relaxation of various central moments to their equilibria under collision. The latter is usually constructed either directly from the Maxwell distribution function or by exploiting its factorization property. We propose a central moment LBM from a different perspective, where its collision operator is constructed by matching the changes in different discrete central moments under collision to the changes in the corresponding continuous central moments as given by the Fokker-Planck (FP) collision model of the Boltzmann equation. The resulting formulation can be interpreted in terms of the relaxation of the various central moments to “equilibria” that depend only on the adjacent, lower order post-collision moments. We designate such newly constructed chain of equilibria as the Markovian central moment attractors and the relaxation rates are based on scaling the drift coefficient of the FP model by the order of the participating moment. We will demonstrate the accuracy and robustness of our new formulation for simulations of a variety of flows using the standard D2Q9 and D3Q27 lattices and also present a comparison against other collision models.   

Nonlocal Coulomb interaction and spin freezing crossover: A route to valence-skipping charge order

Multiorbital systems away from global half-filling host intriguing physical properties promoted by Hund's coupling. Despite increasing awareness of this regime dubbed Hund's metal, effect of nonlocal interaction is still elusive. Here we study a three-orbital model with 1/3 filling (two electrons per site) including the intersite Coulomb interaction V. Using the GW plus extended dynamical mean-field theory, the valence-skipping charge order transition is shown to be driven by V. Most interestingly, the instability to this transition is significantly enhanced in the spin-freezing crossover regime, thereby lowering the critical V to the formation of charge order. This behavior is found to be closely related to the population profile of the atomic multiplet states in the spin-freezing regime. In this regime, maximum spin states are dominant in each total charge subspace with substantial amount of one- and three-electron occupations, which leads to almost equal population of one- and the maximum spin three-electron state. Our finding unveils another feature of the Hund's metal, and has potential implications to the broad range of multiorbital systems as well as the recently discovered charge order in iron-pnictides.   

Improved methods for demonstrating that AKLT systems are gapped

We examine recent advancements in proving the gaps of AKLT systems [1,2], in particular those proposed by Abdul-Rahman et al. [1], which we have later extended so that it can be applied numerically in more general settings. We discuss how this has been used in proving the AKLT gap on a variety of "decorated" lattices in 2D, where the number n of decorating spin-1 sites on each edge of the original lattice is two or larger [3]. Furthermore, we investigate whether the gappedness can be established for several decorated lattices with n=1. We will further explore how the method may be used to demonstrate the existence of the AKLT gap on several uniform lattices without decoration.

[1] H. Abdul-Rahman, M. Lemm, A. Lucia, B. Nachtergaele, and A. Young, arXiv:1901.09297
[2] M. Lemm, A. Sandvik, and S. Yang, arXiv:1904.01043.
[3] N. Pomata and T.-C. Wei, Phys. Rev. B 100, 094429 (2019)