Session N28: Information Theory and Physics

Connecting relevant information to coarse-graining in biological systems

Biological systems must selectively encode partial information about the environment, as dictated by the capacity constraints at work in all living organisms. For example, we cannot see every feature of the light field that reaches our eyes; temporal resolution is limited by transmission noise and delays, and spatial resolution is limited by the finite number of photoreceptors and output cells in the retina. Classical efficient coding theory describes how sensory systems can maximize information transmission given such capacity constraints, but it treats all input features equally. Not all inputs are, however, of equal value to the organism. Our work quantifies whether and how the brain selectively encodes stimulus features, specifically predictive features, that are most useful for fast and effective movements. We have shown that efficient predictive computation starts at the earliest stages of the visual system, in the retina. We borrow techniques from statistical physics and information theory to assess how we get terrific, predictive vision from these imperfect (lagged and noisy) component parts. In broader terms, we aim to build a more complete theory of efficient encoding in the brain, and along the way have found some intriguing connections between formal notions of coarse graining in biology and physics.   

Trading information among multiple variables

Information relevant to cellular function is represented in the concentrations of signaling molecules, including the transcription factors that control gene expression. The amount of relevant information that can be extracted is limited by the precision or information capacity of the mechanisms that respond to changing concentrations.  We have explored this tradeoff in the context of early events in the fruit fly embryo [1] and shown that there are universal features to this tradeoff in limits that are relevant to the embryo [2].  Those results showed that the capacity needed to extract the observed level of positional information in the embryo exceeds what is plausible for a single “readout mechanism’’ or enhancer element.  Here we consider more explicitly how information can be extracted by parallel readout mechanisms, each with limited capacity.  This analysis could lead to a theory for the functional behavior of multiple enhancers in the control of gene expression.  Initial results point to the importance of non-monotonicity and combinatorial responses.

[1] M Bauer et al, Proc Natl Acad Sci (USA) 118, e2109011118 (2021)

[2] M Bauer and W Bialek, arXiv:2212.12471 [physics.bio-ph] (2022).   

Coarse-graining retinal responses to reveal predictive information

The vertebrate retina performs prediction on incoming visual signals, which can compensate for lags in neural processing [1]. This computation is collective, meaning it relies upon interactions between many neurons. However, it is not well understood how correlations between neurons enable prediction in large subpopulations (greater than ten) or when the visual stimulus is complex. In this work, we address these challenges together by searching for maximally-predictive collective variables in large subsets of 93 salamander retinal ganglion cells under stimulation with natural movies. To find these collective variables, we apply a tractable, approximate implementation of the information bottleneck method to neural data [2], and infer a lower-dimensional representation that is maximally informative about the future neural activity. We observe scaling relationships between this mutual information estimate, neural subset size, and information decay timescale. Further, we examine the structure of collective modes learned by this method and compare them to those obtained by other forms of coarse-graining.

[1] S. E. Palmer, O. Marre, M. J. Berry, and W. Bialek, “Predictive information in a sensory population,” Proceedings of the National Academy of Sciences, vol. 112, no. 22, pp. 6908–6913, Jun. 2015

[2] D. E. Gökmen, Z. Ringel, S. D. Huber, and M. Koch-Janusz, “Symmetries and phase diagrams with real-space mutual information neural estimation,” Phys. Rev. E, vol. 104, no. 6, p. 064106, Dec. 2021   

Quantifying information flow in cells

Cells guide their decisions relying on the information that streams through the signalling pathways into the cellular interior. Signalling pathways are interconnected cascades of biochemical reactions which form complex networks. Given the variety of cellular behaviours and responses to external stimuli, the ability to distinguish between a multitude of signals is expected. However, existing quantifications of cellular information flow reported low sensitivity for graded signals. Here, we study information flow in MAPK pathway, one of key signalling pathways in eukaryotes. Combining optogenetic experiments and data analysis based on information theory, we quantify the input-output relationships and elucidate the role of intracellular and extracellular noise, stochastic activations of the pathway and the temporal aspect of the information processing in the cell. We show that the signalling pathway has a higher information capacity than previously reported, as well as bring to light the spatio-temporal aspects of information flow in a cellular popularion.   

Coordination Between Sub-populations of Interneuron in the Spinal Cord Revealed by Information Theory

Understanding the operation of the nervous system is impossible without knowing how different types of neurons coordinate their responses. Just like the brain, the spinal cord consists of many different neuronal types, whose function, input/out connectivity and electrophysiological properties remain poorly characterized. We use information theory to predict how subpopulations of spinal inhibitory and excitatory interneurons may coordinate their responses to their synaptic inputs to achieve maximal information transmission. The theory makes two predictions: 1) there is a positive correlation between threshold (IC50) and noise (inverse slope) of a neuron’s input-output function; 2) below some critical noise value, neuronal populations are expected to further split into sub-types. We verify the first prediction across inhibitory V1, and excitatory V2a and V3 interneuron populations. The second prediction was used to identify subtypes within both V2a and V3 interneurons, which coordinate their input-output behavior in a manner consistent with encoding maximal information given the synaptic inputs.   

Estimating Mutual Information with the Deep Variational Symmetric Information Bottleneck

Mutual Information (MI) captures nonlinear statistical relations between two variables. MI has proved to be useful in analysis of complex systems, in methods involving clustering, feature selection, and dimensionality reduction, among others. Estimation of MI between high-dimensional variables is a challenging task, often requiring impractically large sample sizes for accurate estimation, and thus limiting wider adoption of methods. One approach to resolve the sampling issue is to reduce the dimensionality of the variables. However, such a reduction can destroy correlations between the variables. We resolve this problem using the Deep Variational Symmetric Information Bottleneck (DVSIB), which simultaneously compresses the variables X and Y into two corresponding lower dimensional latent variables Z_X and Z_Y, while maximizing the information between the latent variables. The information between Z_X and Z_Y produced by DVSIB can be used as a proxy for the information between X and Y. We demonstrate the effectiveness of this method by assessing its performance on synthetic and real datasets, showcasing its robustness and accuracy. We show that our method can estimate mutual information between two high-dimensional variables in many cases where standard estimators fail.   

The information bottleneck learns spectral properties of dynamical systems

A common task across the physical sciences is that of model reduction: given a high-dimensional and complex description of a full system, how does one reduce it to a small number of important collective variables? Here we investigate model reduction for dynamical systems using the information bottleneck framework. We show that the optimal compression of a system's state is achieved by encoding spectral properties of its transfer operator. After demonstrating this in analytically-tractable examples, we show our findings hold also in variational compression schemes using experimental fluids data. These results shed light into the latent variables in certain neural network architectures, and show the practical utility of information-based loss functions.   

Obtaining machine learned surrogate underwater acoustics models to enable model reduction

Of great interest in underwater acoustics is the inverse problem to infer environmental seafloor parameters from acoustic data in a shallow ocean environment. One of the primary challenges with this "geoacoustic inversion" is model selection. Complex geoacoustic models have the potential of overfitting, and may include parameters that are unidentifiable, or not constrained by acoustic data. Information geometry provides tools for parameter identifiability analysis, including computational differential geometry tools like the Manifold Boundary Approximation Method (MBAM) for algorithmically finding unidentifiable model parameters to remove. These computational differential geometry tools require the ability to calculate accurate derivatives of model predictions with respect to model parameters. Automatic differentiation (AD) enables rapid evaluation of derivatives but faces implementation challenges with many underwater sound propagation models, such as use with models implemented in "legacy code" and having non-differentiable points in function space. We propose methods for obtaining machine learned (ML) surrogates of the model to which AD can be easily applied. Using a computational underwater acoustics model, surrogate and original model predictions will be compared. Additionally, the use of MBAM for model reduction will be demonstrated to enable next steps for geoacoustic inversion.   

Extending the Manifold Boundary Approximation Method to reduce large-scale, multi-parameter models

The Manifold Boundary Approximation Method (MBAM) is a model reduction technique based on information geometry and sloppy model analysis. This approach interprets a multi-parameter model as a manifold with parameters as coordinates. The Fisher Information Matrix is a natural metric on this manifold, so that distance is statistical distinguishability from data. Multi-parameter models often exhibit a systematic compression of the parameter space in the information metric, so the model manifold is very narrow in most directions, a phenomenon known as sloppiness. We empirically observe that the boundaries of these manifolds are physically interpretable, reduced-order models. MBAM identifies reduced-order models using geodesics to connect a complicated model to a simpler one on the boundary. This approach is computationally and manually intensive, limiting it to moderately-sized models with a few dozen parameters. I present a computationally efficient generalization of MBAM, applicable to models with many more parameters. After reparameterizing, I recast the model reduction problem as a sequence of convex optimizations that can be solved efficiently for high-dimensional parameter spaces. I demonstrate on models from physics, biology, and power systems.   

Information Theoretic Characterization of Critical Phenomena in the Ising Model and Flocking Models

The Ising Model, with its known analytical solutions in one and two dimensions, has been used as a prototypical case study for analyzing critical phenomena and phase transitions. Information Theory has been foundational in analyzing the regularity and patterns contained within random processes, exposing the Informational Architecture instantiated in the underlying physical system. In this work we characterize aspects of the informational architecture of the Ising model on the square lattice using multiple measures, including configurational entropy, configurational complexity, and transfer entropy. Their extremal behavior is compared and contrasted with well understood critical behavior in the Ising model as well as Viscek's flocking model. These results strengthen the foundation for understanding how criticality affects information storage and processing in systems that may not have a clear cut order parameter.   

Random-Energy Secret Sharing via Extreme Synergy

The random-energy model (REM), a solvable spin-glass model, has impacted an incredibly diverse set of problems, from protein folding to combinatorial optimization to many-body localization. Here, we explore a new connection to secret sharing. We formulate a secret-sharing scheme, based on the REM, and analyze its information-theoretic properties. Our analyses reveal that the correlations between subsystems of the REM are highly synergistic and form the basis for secure secret-sharing schemes. We derive the ranges of temperatures and secret lengths over which the REM satisfies the requirement of secure secret sharing. We show further that a special point in the phase diagram exists at which the REM-based scheme is optimal in its information encoding. Our analytical results for the thermodynamic limit are in good qualitative agreement with numerical simulations of finite systems, for which the strict security requirement is replaced by a tradeoff between secrecy and recoverability. Our work offers a further example of information theory as a unifying concept, connecting problems in statistical physics to those in computation.   

A Fourier Tour of Protein Function Prediction

Predicting the biological functions of proteins from their amino acid sequences is one of the long-standing challenges in biology. A comprehensive solution has remained elusive due to the vastness of the combinatorial space of sequences and our limited ability to probe the space experimentally. In this talk, we view protein function prediction from a signal recovery and information theory perspective through the lens of the Fourier transform—also known as Walsh-Hadamard (WH) transform for sequence functions. We discuss how WH transform allows us to view protein functions as a multilinear polynomial and in terms of high-order sparse nonlinear interactions. We demonstrate that an intuitive divide-and-conquer strategy can find the polynomial using a number of samples and times that grows only linearly with the length of the protein sequence. Next, we discuss how we can leverage natural assumptions about the polynomial, such as sparsity, to develop efficient protein function prediction algorithms rooted inc oding theory.