Session K02: Neural Systems III

Associative Memory of Knowledge Structures and Sequences

A long standing challenge in biological and artificial intelligence is to understand how new knowledge can be constructed from known building blocks in a way that is amenable for computation by neuronal circuits. Here we focus on the task of storage and recall of structured knowledge in long term memory. Specifically, we ask how recurrent neuronal networks can store and retrieve multiple knowledge structures. We model structures as a set of binary relations between events and cues (cues may represent e.g., temporal order, spatial location, role in semantic structure). We use binarized holographic reduced representation (HRR) to map such structures to distributed neuronal activity patterns. We then use associative memory plasticity rules to store these activity patterns as fixed points in the recurrent network. By a combination of signal-to-noise analysis and numerical simulations we demonstrate that our model allows for an efficient storage of these knowledge structures, such that the memorized structures as well as their individual building blocks (e.g., events and cues) can be subsequently retrieved, from partial retrieving cues. We show that long-term memory of structured knowledge relies on a new principle of computation beyond the memory basins. Finally, we show that our model can be extended to store sequences as single attractors.   

Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass

Artificial neural networks customarily rely on the backpropagation scheme, where the weights are updated based on the error-function gradients and are propagated from the output to the input layer. Although this approach has proven effective in a wide domain of applications, it lacks a biological rationale in many regards, including the weight symmetry problem, the learning dependence on non-local signals, the freezing of neural activity during the error propagation, and the update locking problem. Alternative training schemes, such as sign symmetry, feedback alignment, and direct feedback alignment, have been introduced, but invariably rely on a backward pass that hinders the possibility of solving all the issues simultaneously.  Here, we propose to replace the backward pass with a second forward pass in which the input signal is modulated based on the network's error. We show that this novel learning rule comprehensively addresses all the above-mentioned issues and applies to both fully connected and convolutional models on MNIST, CIFAR-10, and CIFAR-100. Overall, our work is an important step towards the incorporation of biological principles into machine learning.   

Working memory via combinatorial persistent states atop chaos in a random multivariate network

Working memory (WM) lets us retain information over seconds, but its neural basis is poorly understood. While random networks yield chaotic dynamics reflecting observed irregular brain activity, it's unclear how this could support WM. Here we help reconcile chaos and WM via a network of N units with Q-dimensional activations, where each unit's activation is a weighted sum of all units' previous activations normalized by softmax; weights are i.i.d. Gaussian with variance GQ/N. For large (G, N) network activity is chaotic yet for large Q stably clings to only K of the Q "local" dimensions, enabling it to persist in one of Q-choose-K macrostates atop distributed chaotic microstate dynamics. We show analytically how this symmetry breaking emerges and find that K < 11 in the large G, Q, N limit, bounding the number of macrostates at Q-choose-10. Adding a mean weight, however, lets us increase the number of macrostates to Q-choose-Q/2, letting us employ the network as a distributed WM system similar to a Bloom filter, into whose dynamics we can write several "items" and later query them. This work thus reveals a chaotic network that can nonetheless retain any of a combinatorially large number of memories, shedding new light on how distributed dynamics could support WM.   

Structured Neural Codes Enable Sample Efficient Learning Through Code-Task Alignment

Learning from a limited number of experiences requires suitable inductive biases. To identify how inductive biases are implemented in and shaped by neural codes, we study sample-efficient learning of arbitrary stimulus-response maps from arbitrary neural population codes with biologically-plausible readouts. Using the replica method from the physics of disordered systems, we develop an analytical theory that predicts the average case generalization error of the readout as a function of the number of observed examples. Our theory illustrates in a mathematically precise way how the structure of population codes shapes inductive bias and generalization performance during learning. We further illustrate how a match between the code and the task is crucial for sample-efficient learning. We show applications of our theory to visual cortex recordings and to reservoir computing with RNNs.   

Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?

Equivariance has emerged as a desirable property of neural representations of objects subject to identity-preserving transformations that constitute a group, such as translations and rotations. However, the expressivity of a representation constrained by group equivariance is still not fully understood. We address this gap by providing a generalization of Cover's Function Counting Theorem that quantifies the number of linearly separable and group-invariant binary dichotomies that can be assigned to equivariant representations of objects. We find that the fraction of separable dichotomies is determined by the dimension of the space that is fixed by the group action. We show how this relation extends to operations such as convolutions, element-wise nonlinearities, and global and local pooling. While other operations do not change the fraction of separable dichotomies, local pooling decreases the fraction, despite being a highly nonlinear operation. Finally, we test our theory on intermediate representations of randomly initialized and fully trained convolutional neural networks and find perfect agreement. These results shed light on biological and artificial neural representations that are equivariant to input transformations.   

Signal representation and learning in random feedback neural networks

A fundamental feature of complex biological systems is the ability to form feedback interactions with their environment. A prominent model for studying such interactions is reservoir computing, where learning acts on low-dimensional bottlenecks. Despite the simplicity of this learning scheme, the factors contributing to or hindering the success of training in reservoir networks are in general not well understood. In this work, we study nonlinear feedback networks trained to generate a sinusoidal signal, and analyze how learning performance is shaped by the interplay between internal network dynamics and target properties. By performing exact mathematical analysis of linearized networks, we predict that learning performance is maximized when the target is characterized by an optimal, intermediate frequency which monotonically decreases with the strength of the internal reservoir connectivity. At the optimal frequency, the reservoir representation of the target signal is high-dimensional, desynchronized, and thus maximally robust to noise. We show that our predictions successfully capture the qualitative behavior of performance in nonlinear networks. Moreover, we find that the relationship between internal representations and performance can be further exploited in trained nonlinear networks to explain behaviors which do not have a linear counterpart. Our results indicate that a major determinant of learning success is the quality of the internal representation of the target, which in turn is shaped by an interplay between parameters controlling the internal network and those defining the task.   

Understanding multi-pass stochastic gradient descent via dynamical mean-field theory

Artificial neural networks trained via stochastic gradient descent (SGD) have achieved impressive performances. A general consensus has arisen that understanding SGD successful optimization requires a detailed description of the dynamical trajectory traversed during training. Yet this task is highly nontrivial given that SGD follows a nonequilibrium dynamics in a high-dimensional non-convex loss landscape. Thus, the practical success of SGD is still largely unexplained. We have applied dynamical mean-field theory to derive a full description of the learning curves of SGD and of its performances in prototypical models. This is the first work tracking the high-dimensional dynamics of SGD in the realistic case where the network reuses the available examples multiple times. We have also investigated how different sources of algorithmic noise affect the performance. Comparing SGD to gradient descent in an intrinsically hard problem (phase retrieval) we have shown that SGD noise is key to find good solutions. We have found that an effective fluctuation-dissipation theorem characterizes the stationary dynamics of SGD, extracting the related effective temperature as a function of the hyperparameters. These results point out a novel analogy between SGD and active and driven systems.   

Nested canalizing functions minimize sensitivity and simultaneously promote criticality

We prove that nested canalizing functions are the minimum-sensitivity Boolean functions for any given activity ratio and we characterize the sensitivity boundary which has a nontrivial fractal structure. We further observe, on an extensive database of regulatory functions curated from the literature, that this bound severely constrains the robustness of biological networks. Our findings suggest that the accumulation near the "edge of chaos" in these systems is a natural consequence of a drive towards maximum stability while maintaining plasticity in transcriptional activity.