Session H52: Machine Learning in Nonlinear Physics and Mechanics

A case study in neural networks for scientific data: generating atomic structures

Expertise in both the scientific domain of interest and deep learning techniques are essential in order to properly translate scientific problems into tasks amenable to deep learning. Scientific data has a lot of context; for example, the laws of physics obey certain symmetries. Can the network learn this context from the data or should we impose this context as constraints in our network or training procedures? Additionally, scientific data representations, neural network operations and appropriate loss functions desirable for scientific applications can be very different from those most prevalent in the deep learning literature; when is it appropriate to use existing methods and when is it necessary to develop new ones?

We present examples of these challenges when applying deep learning techniques to the generation of atomic systems (new atomic arrangements that may be crystals, molecules, nanoclusters, polymers, proteins, etc.). We present a novel rotation-equivariant convolutional neural network -- or tensor field network -- that has the ability to articulate, recognize and differentiate local and global features in any orientation in complex atomic systems. We discuss strategies for generating hypothetical atomic structures using the concepts of geometric motifs (the recurring patterns of atoms in materials) and neural networks that can manipulate discrete geometry. We present the use of toy models to test the expressiveness and accuracy of tensor field network operations.   

A computational model for crumpled thin sheets to complement data-driven machine learning

Crumpling is ubiquitous across length scales and diverse structures in nature, yet a complete theoretical description of the mechanisms underlying ridge formation remains elusive. To characterize the intricate damage networks of crumpled thin sheets, our recent work has shown that appropriate simulations can assist data-driven machine learning to overcome the scarcity of high-quality experimental data. Inspired by this data augmentation approach, here we detail a computational model for thin, viscoelastic sheets and demonstrate its ability to capture the properties and behavior of crumpled sheets. We validate the model’s robustness through statistical comparison with high-quality experimental data, and discuss the prospects for its application in assisting data-driven machine learning.   

Machine Learning in a data-limited regime: Augmenting experiments with synthetic data uncovers order in crumpled sheets

Machine learning is a powerful tool for uncovering structure in complex, high-dimensional data. However, a large amount of data is necessary in order to properly train a machine learning network, making it difficult to apply to experimental systems where data is limited. Here we resolve this difficulty by augmenting an experimental dataset with synthetically generated data from a simpler relevant system. Specifically, we study the local order in crease patterns of crumpled sheets, a paradigmatic example of spatial complexity. We supplement sparse crumpled experimental data with abundant simulated sheets of synthetic folds. This technique significantly improves the predictive power in a test problem of pattern completion, demonstrating the usefulness of machine learning in experiments where data may be scarce. Additionally, assessing the accuracy of networks trained with varying types of simulated data reveals the relevance of various physical rules to understanding crease patterns.   

Search and design of stretchable graphene kirigami using convolutional neural networks

Making kirigami-inspired cuts in a sheet has been shown to be an effective way to design stretchable materials with metamorphic properties where the 2D shape can transform into complex 3D shapes. However, a systematic understanding on how cutting patterns alter the mechanical properties of the resulting kirigami remains elusive. Here, we use machine learning (ML) to approximate the objective functions, such as yield stress and yield strain, as functions of the cutting pattern[1]. Our approach enables the rapid discovery of graphene kirigami designs that yield extreme stretchability as verified by molecular dynamics (MD) simulations. We find that convolutional neural networks (CNN) can be applied for regression to achieve an accuracy close to the precision of the MD simulations. This approach can then be used to search for optimal designs that maximize elastic stretchability with only 1000 training samples in a large design space of roughly 4,000,000 candidate designs. This example demonstrates the power and potential of ML in finding optimal kirigami designs at a fraction of iterations that would be required of a purely MD- or experiment-based approach, where no prior knowledge of the governing physics is known or available.

Reference: arXiv preprint arXiv:1808.06111   

Clog prediction in granular hoppers using machine learning methods

Grains discharge from a hopper at constant rate, proceeding probabilistically until a stable arch forms. Thomas and Durian (PRL 2015) showed that recasting the characteristic measure of a flow event from the average mass discharged to the fraction of flow microstates that precede (i.e. cause) a clog explains why the former grows as an exponential function of hole diameter, rather than a critical power law. This makes clear that clogs form as the flow brings new microstates into the vicinity of the outlet, which are randomly sampled until a stable arch is found. Characterizing the flow microstates that cause clogs should then better inform a predictive framework. As a first step, Koivisto and Durian (PRE 2017) found that the same statistics governed hoppers in air or submerged in a viscous fluid. This implies that clog formation depends primarily on position degrees of freedom; however, the phase space of grain microstates in a hopper is extremely high-dimensional. Here, we apply deep learning to probe the function space of position microstates in a two-dimensional hopper, to identify and separate out characteristic structures responsible for clogging. Preliminary analysis of a small dataset gives a cross validation success rate of 90%.   

Tracking Topological Defects in 2D Active Nematics Using Convolutional Neural Networks

The motion of topological defects in active nematics has been modeled by a number of hydrodynamic theories that remain to be fully tested, which we will do by measuring defect dynamics using video microscopy and comparing with theory. However, classical image processing techniques are cumbersome, as variations in image quality and defect morphology require fine tuning for each data set and impede high-throughput processing of experimental data. Here, we use a Convolutional Neural Network (CNN) to efficiently and precisely measure the locations of active defects. We construct a deep CNN to train a defect detector to automatically analyze videos of a microtubule-based active nematic. We labeled 8800 images from which we selected 6600 as training dataset and 1700 as testing dataset. To obtain higher precision, we also consider the temporal relation between the location of defects within consecutive frames and train our CNN correspondingly. We compare results obtained wtih CNN with results generated by traditional image processing algorithms.   

Connecting structure and dynamics in a model of confluent cell tissues using machine learning

Cellular motion in dense tissues often consists of neighbor-swapping events or rearrangements. The rearrangements underlie the glassy dynamics and much of the collective dynamics of dense disordered cellular packings. Here we present a machine learning (ML) approach that links the local disordered structure surrounding a cell with the propensity of the cell to rearrange in a Voronoi cell vertex model. ``Softness,'' S, an ML-derived quantity originally introduced to quantify the link between local structure and rearrangements in inert glassy liquids, provides an effective proxy for a cell's probability to rearrange, P_R. The local structural features that determine the softness of a cell are quantified. Decreasing temperature lowers P_R for a given value of S, but the distribution of S also shifts up, opposing the change, leading to previously-observed sub-Arrhenius dynamics. This contrasts with the behavior of Lennard-Jones glassy liquids, where the distribution of S shifts down with decreasing temperature, leading to super-Arrhenius dynamics.   

Design and learning in multi-stable mechanical networks

Systems with multiple stable states have proven essential in biological and engineering contexts to show multiple functions, store memories and so on. In this work we contrast two paradigms for creating systems with specific stable states in elastic mechanical networks: design and learning. In the design framework, all desired stable states are known in advance and thus material parameters can be optimized on a computer. In contrast, our learning framework considers sequential introduction of desired stable states so that material parameters must be incrementally updated to stabilize each additional state. We show that designed states are optimally stable within elastic networks with Hookean springs. However, incremental learning requires springs with strong non-linearity. We interpret such non-linearity as biasing the distribution of strain in these elastic networks to be localized in a sparse subset of springs, much like a Bayesian prior in sparse regression. In this way, we identify principles for practical implementations of learned multi-stability.   

DropNet : A neural network solution to flow instabilities.

Forecasting phenomenon arising from nonlinear dynamical systems is a daunting task. Formal models are rarely analytically tractable and
numerical simulations are computationally expensive and exponentially sensitive to initial conditions. As a result, the details of the physics of those systems far from equilibrium remains elusive; prompting the need for the development of new tools. Neural networks have been at the foreground of the machine learning community for their versatility and ability to capture complex relationships and patterns in data. Further, predictions stemming from these trained networks are efficient since their computational complexity scales as a set of matrix-vector multiplications. However, the prediction accuracy of these models heavily relies on training with extensive dataset and scales accordingly to the number of samples in the latter. Here we propose an approach to rethink the traditional experimental setting in applied physics in favour of a data-driven framework. We focus on the study of fluid-mediated instabilities. We show that one can predict the spatial structure of droplets arising from a Rayleigh-Taylor instability and highlight several techniques to recover significant physical knowledge from the high dimensional models implemented.   

Visualizing probabilistic models and data with Intensive Principal Component Analysis (InPCA)

Unsupervised learning makes manifest the underlying structure of data without curated training and specific problem definitions. However, the inference of relationships between data points is frustrated by the `curse of dimensionality' in high-dimensions. Inspired by replica theory from statistical mechanics, we consider replicas of the system to tune the dimensionality and take the limit as the number of replicas goes to zero. The result is the intensive embedding, which is not only isometric (preserving local distances) but allows global structure to be more transparently visualized. We develop the Intensive Principal Component Analysis (InPCA) and demonstrate clear improvements in visualizations of the Ising model of magnetic spins, a neural network, and the dark energy cold dark matter ({\Lambda}CDM) model as applied to the Cosmic Microwave Background.   

Physical Symmetries Embedded in Neural Networks

Artificial neural networks (ANNs) have become indispensable tools in many machine learning applications and in recent years, ANNs have become an active area of research in the physical sciences. An important consideration for building ANNs for scientific applications is how to incorporate non-negotiable physical constraints. We propose ANNs with embedded physical symmetries including even-odd symmetry, time-reversibility, positivity, energy-momentum conservation, and Galilean invariance. We constrain the weights of the NN so that the physical properties are exactly satisfied by the neural network output. Furthermore, embedding constraints into the NN can drastically reduce the search space thereby providing an efficient deep learning architecture.   

Maximizing thermal efficiency of heat engines using neuroevolutionary strategies for reinforcement learning

Classic control problems such as Mountain Car [1] and Acrobot [2] are based on simple Newtonian physics. Both have been solved previously with reinforcement learning algorithms. Here, we show that reinforcement learning can also be used to solve classical problems in thermodynamics. Using a reinforcement learning method based on genetic algorithms, our software agent can learn to reproduce thermodynamic cycles without prior knowledge of physical laws. We have created a simulated learning environment which models a simple piston, where an agent can activate thermodynamic processes. With this method, we were able to optimize an artificial neural network based policy to maximize the thermal efficiency for several different cases. Depending on the actions available to the agent, different known cycles emerged, including the Carnot, Stirling, and Otto cycles. Importantly, we show an example of how reinforcement learning can be used to aid scientists in finding solutions to problems that have yet to be fully explored. In one of the heat engine environments, we introduced a non-adiabatic process which caused the engine to lose energy. In this case, the agent produced, what is to the best our knowledge, the best solution for the problem.
[1] A. W. Moore, 1990
[2] S. A. Bortoff, 1992