Bulletin of the American Physical Society
APS March Meeting 2022
Volume 67, Number 3
Monday–Friday, March 14–18, 2022; Chicago
Session F09: Physics of Machine Learning I
8:00 AM–11:00 AM,
Tuesday, March 15, 2022
Room: McCormick Place W-180
Sponsoring
Units:
GSNP GDS DCOMP DSOFT
Chair: Yuhai Tu, IBM T. J. Watson Research Center
Abstract: F09.00001 : Toward Statistical Mechanics of Deep Learning*
8:00 AM–8:36 AM
Presenter:
Haim I Sompolinsky
(Hebrew University of Jerusalem)
Authors:
Haim I Sompolinsky
(Hebrew University of Jerusalem)
Qianyi Li
(Biophysics Program, Harvard University)
To derive properties of network weight space after learning we introduce the Back-Propagating Kernel Renormalization (BPKR), which allows for the incremental integration of the network weights layer-by-layer starting from the network output layer and progressing backward until the first layer's weights are integrated out. This procedure allows us to evaluate important network properties, such as its generalization error, the role of network width and depth, the impact of the size of the training set, the effects of weight regularization and learning stochasticity, as well as the emergent neural representations in each layer.
Unlike most statistical mechanical investigations of learning in neural network, the new theory does not make specific assumption about the statistics of the inputs or the desired target; thus it can be applied to realistic data and tasks.
A heuristic extension of the BPKR to nonlinear DNNs with rectified linear units (ReLU) yields surprisingly good fit to numerical simulations for networks with modest depth, in a wide regime of parameters. Extensions, including deep convolutional networks, and interesting families of nonlinear DNNs will be discussed.
*his research is partially supported by the Swartz Program in Theoretical Neuroscience at Harvard,the NIH grant from the NINDS (1U19NS104653) and the Gatsby Charitable Foundation.
Follow Us |
Engage
Become an APS Member |
My APS
Renew Membership |
Information for |
About APSThe American Physical Society (APS) is a non-profit membership organization working to advance the knowledge of physics. |
© 2025 American Physical Society
| All rights reserved | Terms of Use
| Contact Us
Headquarters
1 Physics Ellipse, College Park, MD 20740-3844
(301) 209-3200
Editorial Office
100 Motor Pkwy, Suite 110, Hauppauge, NY 11788
(631) 591-4000
Office of Public Affairs
529 14th St NW, Suite 1050, Washington, D.C. 20045-2001
(202) 662-8700