Bulletin of the American Physical Society
APS March Meeting 2020
Volume 65, Number 1
Monday–Friday, March 2–6, 2020; Denver, Colorado
Session S18: Statistical Physics Meet Machine LearningInvited
|
Hide Abstracts |
Sponsoring Units: GSNP Chair: Yuhai Tu, IBM TJ Watson Research Center Room: 205 |
Thursday, March 5, 2020 11:15AM - 11:51AM |
S18.00001: Beyond Backprop: Different Approaches to Credit Assignment in Neural Nets Invited Speaker: Irina Rish Backpropagation algorithm (backprop) has been the workhorse of neural net learning for several decades, and its practical effectiveness is demonstrated by recent successes of deep learning in a wide range of applications. This approach uses chain rule differentiation to compute gradients in state-of-the-art learning algorithms such as stochastic gradient descent (SGD) and its variations. However, backprop has several drawbacks as well, including the vanishing and exploding gradients issue, inability to handle non-differentiable nonlinearities and to parallelize weight-updates across layers, and biological implausibility. These limitations continue to motivate exploration of alternative training algorithms, including several recently proposed auxiliary-variable methods which break the complex nested objective function into local subproblems. However, those techniques are mainly offline (batch), which limits their applicability to extremely large datasets, as well as to online, continual or reinforcement learning. The main contribution of our work is a novel online (stochastic/mini-batch) alternating minimization (AM) approach for training deep neural networks, together with the first theoretical convergence guarantees for AM in stochastic settings and promising empirical results on a variety of architectures and datasets. |
Thursday, March 5, 2020 11:51AM - 12:27PM |
S18.00002: Renormalization-group flow in neural-network priors Invited Speaker: Sho Yaida Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. In this talk I extend this correspondence to real neural networks with finite widths, yielding non-Gaussian processes as priors. On the way we shall encounter recursive equations that relate distributions of neural activities from lower to higher layers, reminiscent of renormalization-group flow. |
Thursday, March 5, 2020 12:27PM - 1:03PM |
S18.00003: David Schwab Invited Talk
|
Thursday, March 5, 2020 1:03PM - 1:39PM |
S18.00004: Surya Ganguli Invited Talk
|
Thursday, March 5, 2020 1:39PM - 2:15PM |
S18.00005: Self-tuned annealing in deep learning: How neural networks find generalizable solutions Invited Speaker: Yuhai Tu Despite tremendous success of Stochastic Gradient Descent (SGD) algorithm in deep learning, little is known about how SGD finds generalizable solutions in the high-dimensional weight space. By analyzing the SGD-based learning dynamics and the loss function landscape near solutions found by SGD, we discover a counter-intuitive relation between the weight fluctuation and the loss landscape – the flatter the landscape the smaller the weight variance. To explain this inverse variance-flatness relation, we develop a random landscape theory of SGD, which shows that noise strength (effective temperature) in SGD depends inversely on the landscape flatness and thus SGD serves effectively as a self-tuned (landscape-dependent) annealing mechanism to find the generalizable solutions at the flat minima of the loss landscape. Application of these new insights for preventing catastrophic forgetting will also be discussed. |
Follow Us |
Engage
Become an APS Member |
My APS
Renew Membership |
Information for |
About APSThe American Physical Society (APS) is a non-profit membership organization working to advance the knowledge of physics. |
© 2024 American Physical Society
| All rights reserved | Terms of Use
| Contact Us
Headquarters
1 Physics Ellipse, College Park, MD 20740-3844
(301) 209-3200
Editorial Office
100 Motor Pkwy, Suite 110, Hauppauge, NY 11788
(631) 591-4000
Office of Public Affairs
529 14th St NW, Suite 1050, Washington, D.C. 20045-2001
(202) 662-8700