Bulletin of the American Physical Society
APS March Meeting 2023
Volume 68, Number 3
Las Vegas, Nevada (March 5-10)
Virtual (March 20-22); Time Zone: Pacific Time
Session D02: Statistical Physics Meets Machine Learning I
3:00 PM–6:00 PM,
Monday, March 6, 2023
Room: Room 125
Sponsoring
Units:
GSNP DSOFT DBIO GDS
Chair: Yuhai Tu, IBM T. J. Watson Research Center
Abstract: D02.00009 : The Evolution of the Fisher Information Matrix During Deep Neural Network Training*
5:00 PM–5:12 PM
Presenter:
Chase W Goddard
(Princeton University)
Authors:
Chase W Goddard
(Princeton University)
David J Schwab
(The Graduate Center, CUNY)
Recently, deep neural networks (DNNs) have revolutionized nearly every area of machine learning, and their success has challenged our understanding. In particular, DNNs have empirically been shown to generalize well even in the overparameterized regime. Some correlates of generalization have been found, including flatness of the loss function (Jiang et. al. 2019), and these have even been shown to be causally useful in improving generalization (Foret et. al. 2021), but further study is required. Here, we study the evolution of the Fisher Information Matrix throughout training in both the early and late phase, and identify a number of dynamical signatures of its behavior. While the Fisher often coincides with flatness-based measures such as the Hessian late in training, during the early phase of training they will not in general align. In addition, the Fisher does not require labeled data to compute, allowing its computation on held-out test data. Our method is able to compute the exact Fisher and its eigendecomposition on various subsets of data throughout training, as often as every step along the training curve. In particular, we study the evolution of the Fisher across various dataset splits: train/test, per class, and per domain (in the out-of-distribution setting), and correlate these measures with generalization, both in and out of distribution.
*CWG and DJS were supported by the NSF through the CPBF (PHY-1734030). DJS was also supported by a Simons Fellowship in MMLS and a Sloan Foundation Fellowship.
Follow Us |
Engage
Become an APS Member |
My APS
Renew Membership |
Information for |
About APSThe American Physical Society (APS) is a non-profit membership organization working to advance the knowledge of physics. |
© 2024 American Physical Society
| All rights reserved | Terms of Use
| Contact Us
Headquarters
1 Physics Ellipse, College Park, MD 20740-3844
(301) 209-3200
Editorial Office
100 Motor Pkwy, Suite 110, Hauppauge, NY 11788
(631) 591-4000
Office of Public Affairs
529 14th St NW, Suite 1050, Washington, D.C. 20045-2001
(202) 662-8700