Session L29: Modeling Methods III: Deep Learning and Physics-Informed Learning

The effect of physical constraints on the loss function landscapes of deep learning models

Deep learning models have demonstrated remarkable capabilities at producing fast predictions of complex flow fields. However, incorporating known physics is essential to ensure physically consistent solutions generalize to out-of-sample data. This research investigates the impact of different approaches to impose flow incompressibility and no-penetration boundary conditions on deep learning flow field predictions. This study finds that hard constraints lead to a notably more complex loss landscape, making it more difficult to fit a low-error model. This is compared to the loss-function landscape resulting from a soft constraints approach, where the data loss-function is augmented with additional field and boundary terms, such as in physics-informed neural networks. Finally, the importance of these constraint strategies is studied during extrapolation and prediction of physical quantities, such as lift and drag in an airfoil. This work's findings shed light on the challenges and trade-offs involved in incorporating physics into deep learning models, offering valuable insights for future research in physics-informed machine learning.   

On application of Physics-Informed Neural Networks to Improve Noisy Data of Incompressible Flows

This study explores a novel approach for improving the noisy data of incompressible flow fields by leveraging the Physics Informed Neural Networks (PINNs). Two examples are considered: inviscid flow over a circular cylinder and the 3D axisymmetric Hill Vortex. A neural network is constructed for the spatial variation of the stream-function, to determine the velocity field. Hence, continuity of the flow field is automatically satisfied. Then, the network is trained to minimize the deviation of the constructed flow field from the original field to find the closest divergence-free velocity field for the given flow data. Accordingly, any component, which is not divergence-free (violating continuity) is considered induced by noise and is automatically filtered out. To ensure the accuracy of the corrected data, boundary conditions are introduced into the cost function during training. These boundary conditions guide the filtration process, and help refining the flow field data. One current application of this approach is to improve noisy data obtained from Particle Image Velocimetry (PIV) measurements. This study shows the PINNs' potential in denoising flow fields via incorporating physical knowledge into neural network-based modeling.   

Explainable deep learning for fluid dynamics using a Fourier-wavelet analysis framework

We introduce a new framework that combines the spectral (Fourier) analyses of NNs and nonlinear physics, and leverages recent advances in theory and applications of deep learning, to move toward rigorous analysis of deep NNs for applications involving dynamical systems such as turbulent flows. We will use examples from subgrid-scale modeling of 2D turbulence and Rayleigh-Bernard turbulence, weather forecasting, and modeling gravity waves to show how this framework can be used to systematically address challenges about explainability, generalizability, and stability. For example, the framework shows that in many of such applications, millions of learned parameters in deep convolutional NNs reduce to a few classes of known spectral filters, such as low-pass and Gabor wavelets.   

Reduced-order modeling of fluid flows with transformers

Reduced-order modeling (ROM) of fluid flows has been an active area of research for several decades. The huge computational cost of direct numerical simulations has motivated researchers to develop more efficient alternative methods, such as ROMs and other surrogate models. Similar to many application areas, such as computer vision and language modeling, machine learning and data-driven methods have played an important role in the development of novel models for fluid dynamics. The transformer is one of the state-of-the-art deep learning architectures that has made several breakthroughs in many application areas of artificial intelligence in recent years, including but not limited to natural language processing, image processing, and video processing. In this work, we investigate the capability of this architecture in learning the dynamics of fluid flows in a ROM framework. We use a convolutional autoencoder as a dimensionality reduction mechanism and train a transformer model to learn the system's dynamics in the encoded state space. The model shows competitive results even for turbulent datasets.   

Development of reduced order modeling-based linear system extracting method for efficient data handling with a minimal nonlinearity

Flexible control of fluid flow phenomena is not only of scientific interest but also of engineering importance. However, its high degrees of freedom and strong nonlinearity pose challenges for designing control laws. A solution extensively studied is the application of reduced-order modeling (ROM), which efficiently handles high-dimensional data. In particular, one of the machine learning-based order reduction methods called autoencoder (AE) has attracted attention, leading to various AE-based analysis methods. This is achieved by its ability to map high-dimensional data into a low-dimensional space. However, even with the AE-based ROM, another problem still remains; namely, the extracted low-dimensional features still exhibit strong nonlinearity. Hence, we have investigated a linear system extraction autoencoder (LEAE), which improves the capability of AE to extract a complete linear system from fluid flow phenomena. In this study, we propose an enhanced LEAE, i.e., a partially nonlinear LEAE, using a scheme of time variation of the orbit radius to freely adapt to the time evolution of the latent variables with the flow development. The model extracts a system that can represent the temporal evolution of latent variables by targeting continuously changing flow fields. To achieve this, we focus on both 1) transient and 2) steady flows around a circular cylinder at Re_D=100. Finally, we assess the extracted linear system with minimal nonlinearity and demonstrate its effectiveness.   

Residual-based physics-informed transfer learning (RePIT) strategy to accelerate unsteady fluid flow simulations

Despite the rapid advancements in the performance of central processing units (CPUs), the simulation of unsteady heat and mass transfer is computationally very costly, particularly in large domains. While a big wave of machine learning (ML) has propagated in accelerating computational fluid dynamics (CFD) studies, recent research has revealed that it is unrealistic to completely suppress the error increase as the gap between the training and prediction times increases in single training approach. In this study, we propose a residual-based physics-informed transfer learning (RePIT) strategy to accelerate unsteady heat and mass transfer simulations using ML-CFD cross computation. Our hypothesis is that long-term CFD simulations become feasible if continuous ML-CFD cross computation is periodically carried out to not only reduce increased residuals but also update network parameters with the latest CFD time-series data (transfer learning approach). The cross point of ML-CFD is determined using methods similar to first-principles calculations (physics-informed manner). The feasibility of the proposed strategy was evaluated based on natural convection simulation and compared to the single training approach. In the single training approach, a residual scale change occurred around 100 timesteps leading to some variables exhibiting trends completely opposite to the ground truth. Conversely, it was confirmed that the RePIT strategy maintained the continuity residual within the set range and showed good agreement with the ground truth for all variables and locations. In other words, the RePIT strategy with a grid-based network model does not compromise simulation accuracy for computational acceleration. The simulation was accelerated by 2.5 times, including the parameter-updating time. Open-source CFD software OpenFOAM and open-source ML software TensorFlow were used in this study. In conclusion, this strategy has the potential to significantly reduce the computational cost of CFD simulations while maintaining high accuracy.   

Improving Neural Operators with Physics Informed Token Transformers

Solving Partial Differential Equations (PDEs) is the core of many fields of science and engineering. While classical approaches are often prohibitively slow, machine learning models often fail to incorporate complete system information. Over the past few years, neural operators have shown significant promise in PDE surrogate modeling. However, neural operators often do not incorporate physically relevant information into the learning process, namely the governing equations. To address this issue, we introduce PITT: Physics Informed Token Transformer, that uses a transformer-based architecture on top of existing, popular neural operators. The purpose of PITT is to incorporate the knowledge of physics by embedding partial differential equations (PDEs) into the learning process. PITT uses an equation tokenization method to learn an analytically-driven numerical update operator. By tokenizing PDEs and embedding partial derivatives, the transformer models become aware of the underlying knowledge behind physical processes. Combining neural operators with transformers offers a powerful and flexible method to incorporate physics knowledge into the neural operator learning process. To demonstrate this, PITT is tested on challenging 1D and 2D PDE neural operator prediction tasks. The results show that PITT outperforms popular neural operator models and has the ability to extract physically relevant information from governing equations.   

Dimensional compression and reconstruction for unstructured finite volume meshes via geometric deep learning

The finite volume method (FVM) is an attractive approach to simulate complex physical phenomena by solving integral forms of the governing physical equations. The FVM often employs an unstructured mesh (UM) to spatially discretize the domain into a mesh of cells that are not created in the form of a structured grid. This lack of a natural grid structure for an UM makes the direct application of convolutional neural networks for the purpose of model reduction currently untenable. The present work aims to overcome this limitation by incorporating graph neural networks (GNNs) to perform dimensional compression intuitively upon an UM in a machine learning framework. GNNs are a class of machine learning methods selected for this application due to their ability to represent and extract information from relational data, as is seen in an UM. A GNN-based approach to perform dimensional compression and reconstruction upon the FVM employing an UM will be presented, and the method will be tested on a problem with a Kelvin-Helmholtz instability.   

Physics-informed neural network for enhancement of weather forecasts

The significance of accurate weather prediction has become more relevant in recent years. Currently, weather models primarily rely on historic data statistics and numerical methods. However, the emergence of artificial intelligence offers new possibilities for addressing the demand for accurate information on short-to-mid-term weather events. One particular field in which this is crucial concerns airports' vicinities, where it is vital for air control management to be constantly informed on potential severe weather conditions that could affect the airport operation. Accurate predictions can lead to significant cost savings by enabling efficient flight planning and optimal allocation of operational resources. To achieve these goals, operators require predictions with look-ahead times of at least one hour in addition to high spatial resolution. Our research focuses on leveraging physics-informed neural networks (PINNs) to precisely reconstruct the weather field from limited data provided by weather stations on a finer spatially-resolved grid. By enforcing compliance with physics constraints, we can enhance the deterministic and comprehensive reconstruction of the field data (i.e. wind velocity and pressure), enabling better anticipation of weather event's time evolution.   

Using self-adaptive physics-informed learning to estimate orographic gravity waves

Despite the continuing increase of computing power, the multi-scale nature of geophysical fluid dynamics implies that many important physical processes are still represented using physical parameterization. This traditional approach exhibits persistent and systematic shortcomings due to an inadequate representation of unresolved processes and remains impractical to employ when the boundary and/or initial conditions are not well-defined. In this context, deep learning models are considered an attractive alternative approach as they offer the potential of generalizing the solution while still being able to respect physical constraints. The starting point of the present work is to use Physics-Informed Neural Networks (PINNs) to estimate orographic gravity wave parameters. A fixed budget online adaptive learning strategy is proposed to improve the performance of PINNs by correcting adaptively the distribution of unsupervised training points during the training process. This strategy is shown to accurately capture important couplings between meteorological variables, especially in the vicinity of the mountain. The numerical results also demonstrate the capability of PINNs for solving inverse problems, i.e. estimating dimensionless parameters related to the flow (Richardson and Reynolds numbers) or the shape of the mountain from a downsampled gravity wave field.   

Machine-learned reduced order modeling toward an effective flow control framework

Reduced order modeling is one of the promising techniques for designing efficient flow control schemes. However, mathematical derivation of a control law is still difficult if the low-dimensionalized dynamics is nonlinear. We propose a new machine-learned reduced order modeling to derive linear ordinary differential equations (ODE) that govern low-dimensionalized flow dynamics, named linear system extraction autoencoder (LEAE). The LEAE consists of a convolutional neural network-based autoencoder (CNN-AE) and an additional layer in its bottleneck. The CNN-AE has ever been utilized to efficiently map a high-dimensional phenomena into a low-dimensional latent space, and here we also employ the additional layer named linear ODE (LODE) layer to seek a governing equation of the latent dynamics in a form of linear ODE. Inside the LODE layer, a time integration scheme of the ODE for one time step is emulated, and the coefficient matrix is optimized through the training process with temporally consecutive latent vectors. It should be emphasized that the CNN-AE and the LODE layer are trained simultaneously such that the latent dynamics are governed by a system of linear ODE. This LEAE can successfully reproduce the two-dimensional cylinder wake at Re_D = 100 as an example of high-dimensional flow data. In the talk, we will show statistical assessments and applications to cases with blowing/suction control.   

A Data-Free Partial Differential Equation Solver Based on Physics-Informed Neural Networks (PINN): FDM-PINN

Solving partial differential equations (PDEs) based on machine learning methods has attracted significant attention recently. One of the representatives is Physics-informed Neural Network (PINN), which is based on deep learning and incorporates physical laws into its loss function. However, PINN has limited performance when the training data vanishes, leading to a challenge in developing a data-free PDE solver in the framework of PINN. In this talk, we propose a novel data-free PDE solver (FDM-PINN), which combines the superiorities of both FDM and PINN. The total loss function in FDM-PINN only contains one term related to a set of mesh-based difference equations, which allows that the boundary condition (BC) and initial condition (IC) are imposed exactly instead of treating them as parts of the loss function in the original PINN. Furthermore, auto-differential (AD) technique in original PINN, which has lower accuracy verified by other researchers, has been replaced by finite difference (FD) technique when modeling the derivative terms. We demonstrate the advantages of FDM-PINN over other PINN-based methods quantitatively through examples of various PDE types.