Bulletin of the American Physical Society
APS March Meeting 2022
Volume 67, Number 3
Monday–Friday, March 14–18, 2022; Chicago
Session K02: Neural Systems IIIFocus Recordings Available

Hide Abstracts 
Sponsoring Units: DBIO Chair: Philipp Fleig, University of Pennsylvania Room: McCormick Place W175C 
Tuesday, March 15, 2022 3:00PM  3:36PM 
K02.00001: Associative Memory of Knowledge Structures and Sequences Invited Speaker: Julia A Steinberg A long standing challenge in biological and artificial intelligence is to understand how new knowledge can be constructed from known building blocks in a way that is amenable for computation by neuronal circuits. Here we focus on the task of storage and recall of structured knowledge in long term memory. Specifically, we ask how recurrent neuronal networks can store and retrieve multiple knowledge structures. We model structures as a set of binary relations between events and cues (cues may represent e.g., temporal order, spatial location, role in semantic structure). We use binarized holographic reduced representation (HRR) to map such structures to distributed neuronal activity patterns. We then use associative memory plasticity rules to store these activity patterns as fixed points in the recurrent network. By a combination of signaltonoise analysis and numerical simulations we demonstrate that our model allows for an efficient storage of these knowledge structures, such that the memorized structures as well as their individual building blocks (e.g., events and cues) can be subsequently retrieved, from partial retrieving cues. We show that longterm memory of structured knowledge relies on a new principle of computation beyond the memory basins. Finally, we show that our model can be extended to store sequences as single attractors. 
Tuesday, March 15, 2022 3:36PM  4:12PM 
K02.00002: Errordriven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass Invited Speaker: Giorgia Dellaferrera Artificial neural networks customarily rely on the backpropagation scheme, where the weights are updated based on the errorfunction gradients and are propagated from the output to the input layer. Although this approach has proven effective in a wide domain of applications, it lacks a biological rationale in many regards, including the weight symmetry problem, the learning dependence on nonlocal signals, the freezing of neural activity during the error propagation, and the update locking problem. Alternative training schemes, such as sign symmetry, feedback alignment, and direct feedback alignment, have been introduced, but invariably rely on a backward pass that hinders the possibility of solving all the issues simultaneously. Here, we propose to replace the backward pass with a second forward pass in which the input signal is modulated based on the network's error. We show that this novel learning rule comprehensively addresses all the abovementioned issues and applies to both fully connected and convolutional models on MNIST, CIFAR10, and CIFAR100. Overall, our work is an important step towards the incorporation of biological principles into machine learning. 
Tuesday, March 15, 2022 4:12PM  4:24PM 
K02.00003: Robust sequential retrieval of memories in interactionmodulated neural networks Lukas Herron, BingKan Xue, Pablo Sartori Associative memory networks are capable of storing a set of neural configurations and retrieving one of these memories given an initial state, a classic example being the Hopfield network. Such networks have been generalized to perform sequential retrieval, i.e., to dynamically retrieve a prescribed sequence of memories. We identify three requirements for sequential retrieval: a destabilization of the present memory, a directional bias towards the next memory in the sequence, and a separation of timescales. Earlier works have utilized timedelayed nodes to satisfy these requirements. We demonstrate that these models belong to a class where the delayed nodes exert an external input on the nodes of the original network. In contrast, we explore a class of models in which the delayed nodes modulate the interactions between the original nodes. We give several examples of such interactionmodulated networks capable of sequential retrieval. Further, we examine the phase space of sequential retrieval to compare the robustness of inputmodulated and interactionmodulated networks within a unifying framework. 
Tuesday, March 15, 2022 4:24PM  4:36PM 
K02.00004: Working memory via combinatorial persistent states atop chaos in a random multivariate network Rich Pang Working memory (WM) lets us retain information over seconds, but its neural basis is poorly understood. While random networks yield chaotic dynamics reflecting observed irregular brain activity, it's unclear how this could support WM. Here we help reconcile chaos and WM via a network of N units with Qdimensional activations, where each unit's activation is a weighted sum of all units' previous activations normalized by softmax; weights are i.i.d. Gaussian with variance GQ/N. For large (G, N) network activity is chaotic yet for large Q stably clings to only K of the Q "local" dimensions, enabling it to persist in one of QchooseK macrostates atop distributed chaotic microstate dynamics. We show analytically how this symmetry breaking emerges and find that K < 11 in the large G, Q, N limit, bounding the number of macrostates at Qchoose10. Adding a mean weight, however, lets us increase the number of macrostates to QchooseQ/2, letting us employ the network as a distributed WM system similar to a Bloom filter, into whose dynamics we can write several "items" and later query them. This work thus reveals a chaotic network that can nonetheless retain any of a combinatorially large number of memories, shedding new light on how distributed dynamics could support WM. 
Tuesday, March 15, 2022 4:36PM  4:48PM 
K02.00005: Dynamical phases and computation in nonlinear networks with correlated couplings Daniel Wennberg, Atsushi Yamamura, Surya Ganguli, Hideo Mabuchi Recurrent networks with random couplings can serve as minimal models of diverse systems including neural networks, large ecosystems, and annealers for combinatorial optimization. The statistics of fixed points and chaos in different parameter regimes can be studied using random matrix methods and dynamic meanfield theory. To date, such studies have usually assumed network units without intrinsic dynamics beyond linear relaxation, and treatments with nonlinear selfcouplings have only considered uncorrelated crosscouplings. We extend these methods to networks with both nonlinear selfcouplings and nonzero coupling covariances, and analyze the disintegration of the energy landscape and transition to chaos as the network is tuned away from symmetric couplings. We study the computational relevance of different dynamical phases for the operation of modern neuromorphic annealing hardware, quantifying both the implications of unwanted asymmetric coupling noise and the potential for advantageous chaotic annealing via a tunable coupling asymmetry. Along the way, we derive selfconsistent equations for the spectrum of the most general class of large Gaussian random matrices. 
Tuesday, March 15, 2022 4:48PM  5:00PM 
K02.00006: Structured Neural Codes Enable Sample Efficient Learning Through CodeTask Alignment Blake Bordelon, Cengiz Pehlevan Learning from a limited number of experiences requires suitable inductive biases. To identify how inductive biases are implemented in and shaped by neural codes, we study sampleefficient learning of arbitrary stimulusresponse maps from arbitrary neural population codes with biologicallyplausible readouts. Using the replica method from the physics of disordered systems, we develop an analytical theory that predicts the average case generalization error of the readout as a function of the number of observed examples. Our theory illustrates in a mathematically precise way how the structure of population codes shapes inductive bias and generalization performance during learning. We further illustrate how a match between the code and the task is crucial for sampleefficient learning. We show applications of our theory to visual cortex recordings and to reservoir computing with RNNs. 
Tuesday, March 15, 2022 5:00PM  5:12PM 
K02.00007: Capacity of Groupinvariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views? Matthew S Farrell, Blake Bordelon, Shubhendu Trivedi, Cengiz Pehlevan Equivariance has emerged as a desirable property of neural representations of objects subject to identitypreserving transformations that constitute a group, such as translations and rotations. However, the expressivity of a representation constrained by group equivariance is still not fully understood. We address this gap by providing a generalization of Cover's Function Counting Theorem that quantifies the number of linearly separable and groupinvariant binary dichotomies that can be assigned to equivariant representations of objects. We find that the fraction of separable dichotomies is determined by the dimension of the space that is fixed by the group action. We show how this relation extends to operations such as convolutions, elementwise nonlinearities, and global and local pooling. While other operations do not change the fraction of separable dichotomies, local pooling decreases the fraction, despite being a highly nonlinear operation. Finally, we test our theory on intermediate representations of randomly initialized and fully trained convolutional neural networks and find perfect agreement. These results shed light on biological and artificial neural representations that are equivariant to input transformations. 
Tuesday, March 15, 2022 5:12PM  5:24PM 
K02.00008: Signal representation and learning in random feedback neural networks Lee Susman, Francesca Mastrogiuseppe, Naama Brenner, Omri Barak A fundamental feature of complex biological systems is the ability to form feedback interactions with their environment. A prominent model for studying such interactions is reservoir computing, where learning acts on lowdimensional bottlenecks. Despite the simplicity of this learning scheme, the factors contributing to or hindering the success of training in reservoir networks are in general not well understood. In this work, we study nonlinear feedback networks trained to generate a sinusoidal signal, and analyze how learning performance is shaped by the interplay between internal network dynamics and target properties. By performing exact mathematical analysis of linearized networks, we predict that learning performance is maximized when the target is characterized by an optimal, intermediate frequency which monotonically decreases with the strength of the internal reservoir connectivity. At the optimal frequency, the reservoir representation of the target signal is highdimensional, desynchronized, and thus maximally robust to noise. We show that our predictions successfully capture the qualitative behavior of performance in nonlinear networks. Moreover, we find that the relationship between internal representations and performance can be further exploited in trained nonlinear networks to explain behaviors which do not have a linear counterpart. Our results indicate that a major determinant of learning success is the quality of the internal representation of the target, which in turn is shaped by an interplay between parameters controlling the internal network and those defining the task. 
Tuesday, March 15, 2022 5:24PM  5:36PM 
K02.00009: Understanding multipass stochastic gradient descent via dynamical meanfield theory Francesca Mignacco Artificial neural networks trained via stochastic gradient descent (SGD) have achieved impressive performances. A general consensus has arisen that understanding SGD successful optimization requires a detailed description of the dynamical trajectory traversed during training. Yet this task is highly nontrivial given that SGD follows a nonequilibrium dynamics in a highdimensional nonconvex loss landscape. Thus, the practical success of SGD is still largely unexplained. We have applied dynamical meanfield theory to derive a full description of the learning curves of SGD and of its performances in prototypical models. This is the first work tracking the highdimensional dynamics of SGD in the realistic case where the network reuses the available examples multiple times. We have also investigated how different sources of algorithmic noise affect the performance. Comparing SGD to gradient descent in an intrinsically hard problem (phase retrieval) we have shown that SGD noise is key to find good solutions. We have found that an effective fluctuationdissipation theorem characterizes the stationary dynamics of SGD, extracting the related effective temperature as a function of the hyperparameters. These results point out a novel analogy between SGD and active and driven systems. 
Tuesday, March 15, 2022 5:36PM  5:48PM 
K02.00010: Nested canalizing functions minimize sensitivity and simultaneously promote criticality Alkan Kabakcioglu, Hamza Coban We prove that nested canalizing functions are the minimumsensitivity Boolean functions for any given activity ratio and we characterize the sensitivity boundary which has a nontrivial fractal structure. We further observe, on an extensive database of regulatory functions curated from the literature, that this bound severely constrains the robustness of biological networks. Our findings suggest that the accumulation near the "edge of chaos" in these systems is a natural consequence of a drive towards maximum stability while maintaining plasticity in transcriptional activity. 
Follow Us 
Engage
Become an APS Member 
My APS
Renew Membership 
Information for 
About APSThe American Physical Society (APS) is a nonprofit membership organization working to advance the knowledge of physics. 
© 2023 American Physical Society
 All rights reserved  Terms of Use
 Contact Us
Headquarters
1 Physics Ellipse, College Park, MD 207403844
(301) 2093200
Editorial Office
1 Research Road, Ridge, NY 119612701
(631) 5914000
Office of Public Affairs
529 14th St NW, Suite 1050, Washington, D.C. 200452001
(202) 6628700