Bulletin of the American Physical Society
2023 APS March Meeting
Volume 68, Number 3
Las Vegas, Nevada (March 5-10)
Virtual (March 20-22); Time Zone: Pacific Time
Session B12: Statistical Physics Meets Machine Learning
11:30 AM–2:30 PM,
Monday, March 6, 2023
Room: Room 235
Sponsoring
Unit:
GSNP
Chair: Yuhai Tu, IBM T. J. Watson Research Center
Abstract: B12.00003 : A Picture of the Prediction Space of Deep Networks*
12:42 PM–1:18 PM
Presenter:
Pratik Chaudhari
(University of Pennsylvania)
Authors:
Jialin Mao
(University of Pennsylvania)
Itay Griniasty
(Cornell University)
Rubing Yang
(University of Pennsylvania)
Han Kheng Teoh
(Cornell University)
Rahul Ramesh
(University of Pennsylvania)
Mark K Transtrum
(Brigham Young University)
James P Sethna
(Cornell University)
Pratik Chaudhari
(University of Pennsylvania)
I will argue that deep networks generalize well because of a characteristic structure in the space of learning tasks. The input correlation matrix for typical tasks has a “sloppy” eigenspectrum where, in addition to a few large eigenvalues, there is a large number of small eigenvalues that are distributed uniformly over a very large range. As a consequence, quantities such as the Hessian or the Fisher Information Matrix also have a sloppy eigenspectrum. Using these ideas, I will demonstrate an analytical non-vacuous generalization bound for deep networks.
I will argue that training a deep network is computationally tractable because for sloppy tasks, the training process explores an extremely low-dimensional (~0.001% of the dimensionality of the embedding space) manifold in the prediction space. Models with different neural architectures (fully-connected, convolutional, residual, and attention-based), training methods (stochastic gradient descent and variants), weight initializations (random vs. pre-training on random labels), and regularization techniques (weight-decay, batch-normalization, and data-augmentation) evolve along very similar trajectories in the prediction space when trained for the same task and traverse a very similar manifold.
*We would like the acknowledge funding from the National Science Foundation (RI 2145164, CCF 2212519), the Office of Naval Research (N000142212255) and Amazon Web Services.
Follow Us |
Engage
Become an APS Member |
My APS
Renew Membership |
Information for |
About APSThe American Physical Society (APS) is a non-profit membership organization working to advance the knowledge of physics. |
© 2025 American Physical Society
| All rights reserved | Terms of Use
| Contact Us
Headquarters
1 Physics Ellipse, College Park, MD 20740-3844
(301) 209-3200
Editorial Office
100 Motor Pkwy, Suite 110, Hauppauge, NY 11788
(631) 591-4000
Office of Public Affairs
529 14th St NW, Suite 1050, Washington, D.C. 20045-2001
(202) 662-8700