Bulletin of the American Physical Society
APS March Meeting 2022
Volume 67, Number 3
Monday–Friday, March 14–18, 2022; Chicago
Session F09: Physics of Machine Learning I
8:00 AM–11:00 AM,
Tuesday, March 15, 2022
Room: McCormick Place W-180
Sponsoring
Units:
GSNP GDS DCOMP DSOFT
Chair: Yuhai Tu, IBM T. J. Watson Research Center
Abstract: F09.00009 : Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines*
10:00 AM–10:12 AM
Presenter:
Aurélien Decelle
(Universidad Complutense de Madrid)
Authors:
Aurélien Decelle
(Universidad Complutense de Madrid)
Beatriz Seoane
(Univ Complutense)
Cyril Furtlehner
(Inria, Université Paris Saclay)
time due to the difficulty of computing precisely the log-likelihood gradient. Over
the past decades, many works have proposed more or less successful training
recipes but without studying the crucial quantity of the problem: the mixing time.
In this work, we show that this mixing time plays a crucial role
in the dynamics and stability of the trained model, and that RBMs operate in two
well-defined regimes, namely equilibrium and out-of-equilibrium, depending on
the interplay between this mixing time of the model and the number of steps, k,
used to approximate the gradient. We further show empirically that this mixing
time increases with the learning, which often implies a transition from one regime
to another as soon as k becomes smaller than this time. In particular, we show that
using the popular k (persistent) contrastive divergence approaches, with k small,
the dynamics of the learned model are extremely slow and often dominated by
strong out-of-equilibrium effects. On the contrary, RBMs trained in equilibrium
display faster dynamics, and a smooth convergence to dataset-like configurations
during the sampling.
*A.D. was supported by the Comunidad de Madrid and the Complutense University of Madrid (Spain)through the Atraccion de Talento program (Ref. 2019-T1/TIC-13298). B.S. was supported by theComunidad de Madrid and the Complutense University of Madrid (Spain) through the Atraccion deTalento program (Ref. 2019-T1/TIC-12776)
Follow Us |
Engage
Become an APS Member |
My APS
Renew Membership |
Information for |
About APSThe American Physical Society (APS) is a non-profit membership organization working to advance the knowledge of physics. |
© 2023 American Physical Society
| All rights reserved | Terms of Use
| Contact Us
Headquarters
1 Physics Ellipse, College Park, MD 20740-3844
(301) 209-3200
Editorial Office
1 Research Road, Ridge, NY 11961-2701
(631) 591-4000
Office of Public Affairs
529 14th St NW, Suite 1050, Washington, D.C. 20045-2001
(202) 662-8700