Energy-based Sequential Memory Networks at the Adiabatic Limit*

In memory modeling, the energy paradigm has been a cornerstone where the memories are stored as energy minima, with neural dynamics descending along the energy surface to reach a memory state. Recent advancements have unraveled factors affecting memory capacity and offered avenues for improvement. These improved models, with exponential memory capacity, have bolstered state-of-the-art AI memory capabilities in various practical domains. However, their static energy surface confines them to single memory retrieval, unlike human memory's diversity, which has sequential characterisitcs. Here, we introduce a new neural memory model featuring a dynamic energy surface. In the adiabatic limit of energy surface changes, the model exhibits stable sequence memory recall. By leveraging the architectural modifications that boosted singleton memory capacity, we enhance sequence memory capacity. Empirical validation with a 100-neuron model shows a 1000-fold improvement in memory capacity over existing sequence memory models. Our work demonstrates the potential of the temporal energy paradigm in advancing sequence memory modeling.