Session M48: Building the Bridge to Exascale: Applications and Opportunities for Materials, Chemistry, and Biology II

High-Performance Single-Particle Imaging Reconstruction on Pre-Exascale Computing Platforms

Single-particle imaging (SPI) reconstruction is the analysis of X-ray diffraction patterns from a light source directed at biological molecules. SPI uses intense X-ray free electron laser (FEL) pulses to obtain a continuous diffraction pattern from single molecules in a serial manner--one molecule at a time The analysis is computationally intensive, requiring diffraction patterns to be sorted into conformations and oriented in 3D for model building and refinement. SPI is also data-intensive, requiring potentially hundreds of thousands of diffraction patterns in order to produce a high-resolution result.

The ExaFEL project is an Exascale Computing Project with the goal of performing SPI reconstruction in quasi-real time where the analysis keeps up with experimental data rates. Upgraded detectors and light sources such as the Linac Coherent Light Source (LCLS) are delivering SPI experimental data at ever higher velocities and larger volumes.

This talk will overview the capabilities of SpiniFEL, the high-performance analytics code for SPI reconstruction developed by the ExaFEL team. SpiniFEL implements the multi-tiered iterative phasing (M-TIP) algorithm for SPI reconstruction. SpiniFEL has been accelerated via GPU offloading as well as through parallel and distributed programming models. It can run using the Message Passing Interface (MPI) or via the task-based Legion Runtime System. SpiniFEL runs on the pre-exascale machines Summit at the Oak Ridge Leadership Computing Facility (OLCF) and Perlmutter at the National Energy Research Scientific Computing Center (NERSC).

We will present results of SpiniFEL runs on pre-exascale computing platforms and outline how SpiniFEL SPI analysis fits into the larger vision of an inter-facility workflow that moves data from acquisition at an experimental facility to computation at a supercomputing center, so scientists can quickly determine structures and produce high quality results efficiently using scarce experimental resources.   

Scaling graph models for large computational catalysis datasets to thousands of GPUs on Perlmutter

The workhouse of computational catalysis, density functional theory, remains the rate-limiting step in complex and high-fidelity investigations of experimentally-relevant catalyst behavior. Machine learning approaches to accelerate these simulations have seen much progress in the last few years. Large computational catalyst datasets, such as the Open Catalyst 2020 (OC20) dataset, have drastically improved the generalizability of machine learning surrogate models. Most state of the art models on the OC20 dataset are variations of graph neural networks trained on 10s to 100s of GPUs. As one of the NERSC early science application projects (NESAP) for the Perlmutter supercomputer, we are working to scale these models to 1000s of GPUs to identify the scaling behavior of accuracy with model size, and will present scaling results collected during the NESAP project. I will also discuss challenges in scaling these models and research directions that would be valuable for using these large models in day-to-day science efforts.    

Scalable Solutions for Training Machine Learned Interatomic Potentials

The promise of all machine learning (ML) methods is that model accuracy can in principle be improved indefinitely so long as new training data is provided.  This keeps model predictions as interpolations within the trained space rather than relying on uncertain predictions arising from extrapolations.  For machine learned interatomic potentials used in molecular dynamics, there is no way to know a priori all the states of the material that will be observed in a large-scale production simulation. Automated training data curation either in real-time or diversity maximizing techniques are sought after to alleviate these concerns, though assembled training sets now scale with the size of the computing resources used. This talk will overview the user friendly FitSNAP code and its integration into the Exascale Computing Project EXAALT software stack with a focus on the challenges and advances made to tackle exascale sized training sets needed to construct robust and truly transferable interatomic potentials.   

Massively-Parallel Real-time TDDFT using Plane-wave Pseudopotential Formulation: Application to Studying Electronic Excitation in Solvated DNA.

We discuss the real-time propagation approach to time-dependent density functional theory (RT-TDDFT) in the planewave pseudopotential formulation for simulating electronic excitation and dynamics in complex systems. In particular, our implementation in Qb@ll code is discussed, and we present its application to studying non-equilibrium energy transfer excitation in solvated DNA under ion irradiation. We will discuss how propagating maximally-localized Wannier functions (MLWFs) can provide key insights at the molecular level. We further discuss how hybrid DFT functionals can be implemented efficiently using the time-dependent MLWFs.   

INQ: a state-of-the art implementation of density functional theory for GPUs

In this talk I will present INQ, a new implementation of density functional theory (DFT) and time-dependent DFT (TDDFT) written from scratch to work on graphical processing units (GPUs).

Besides GPU support, INQ makes use of modern code design features and techniques, to make development fast and simple, and to ensure the quality of the program. By designing the code around algorithms, rather than against specific implementations and numerical libraries, we provide a concise and modular code that is simple to understand, flexible, and extensible.

What we achieve is a fairly complete DFT/TDDFT implementation in roughly 12,000 lines of open-source C++ code. It represents a modular platform for community-driven application development on emerging high-performance computing architectures. The code is freely accesible at http://gitlab.com/npneq/inq .

In TDDFT simulations on GPU-based supercomputers INQ achieves excellent performance. It can handle hundreds and thousands of atoms, with simulation times of a second or less per time-step, and scale to thousands of GPUs.   

MDSuite: A post-processing engine for particle simulations.

Particle-based simulations are experiencing a rapid growth wherein system sizes in the hundreds of thousands or even millions are becoming commonplace. With this growth in system size comes the additional challenge of post-processing the simulation data.

In this talk, we introduce the Python package MDSuite. MDSuite is designed for the post-processing of particle-based simulation in an efficient manner and on modern hardware. Built on top of TensorFlow, MDSuite calculators are fully parallelised, gpu-enabled, and, due to the use of modern data pipe-lining methods, completely memory safe. Furthermore, the use of HDF5 and SQL database structures enables effective tracking of calcualtion parameters as well as a compressed trajectory storage medium. We present MDSuite as a standalone package for the storage, analysis, and comparison of large-scale simulation studies.   

Adaptive Numerical Solution of Kadanoff-Baym Equations

A time-stepping scheme with adaptivity in both the step size and the integration order is presented in the context of non-equilibrium dynamics described via Kadanoff-Baym equations. The accuracy and effectiveness of the algorithm are analysed by obtaining numerical solutions of exactly solvable models. We find a significant reduction in the number of time-steps compared to fixed-step methods. Due to the at least quadratic scaling of Kadanoff-Baym equations, reducing the amount of steps can dramatically increase the accessible integration time, opening the door for the study of long-time dynamics in interacting systems. A selection of illustrative examples is provided, among them interacting and open quantum systems as well as classical stochastic processes. An open-source implementation of our algorithm in the scientific-computing language Julia is made available.   

Learning the Exchange-Correlation Functional from Nature with Fully Differentiable Density Functional Theory

We present recent work on the development of a fully-differentiable density functional theory simulator (DQC – Differentiable Quantum Chemistry) where the exchange-correlation functional is represented by a trainable neural network. We demonstrate how the computing paradigm of automatic differentiation, constrained by the Kohn-Sham framework, can provide a novel way to construct highly accurate exchange-correlation functionals using heterogeneous experimental data even for highly limited datasets. Using only eight experimental data points on diatomic molecules, our trained exchange-correlation networks enable improved prediction accuracy of atomization energies across a collection of 104 molecules containing new bonds, atoms, and molecules, not present in the training dataset.    

Electronic Excitations from GW-BSE in Large Molecular Systems via hybrid embeddings

Many-Body Green’s Functions Theory in the GW approximation with the Bethe-Salpeter Equation (GW-BSE) is a post Density-functional theory method to obtain accurate energies of charged and neutral electronic excitations for a wide range of materials from inorganic crystals to organic molecules. However, as GW-BSE scales, depending on implementation details, with the fourth power of the system size, its application to large complex molecular systems, such as polymers, polymer composites, or complex supramolecular assemblies is challenging.

In this talk we showcase some computational strategies based on embedding an active GW-BSE subregion of a complex system in regions described by low level methods as implemented in our VOTCA-XTP package¹. We focus particularly on embedding a GW-BSE model withing a projector based splitting of DFT densities. We consider the effects of eg. basis-set truncations and restrictions to product space basis on the calculated excitations of a few prototypical systems such as solvated dyes.

1. G. Tirimbò, V. Sundaram, O. Çaylak, W. Scharpach, J. Sijen, C. Junghans, J. Brown, F. Zapata Ruiz, N. Renaud, J. Wehner, and B. Baumeier, "Excited-state electronic structure of molecules using many-body Green’s functions: Quasiparticles and electron–hole excitations with VOTCA-XTP", J. Chem. Phys. 152, 114103 (2020) https://doi.org/10.1063/1.5144277