Understanding Optimization in Deep Learning with Central Flows
Jeremy Cohen Alex Damian Ameet Talwalkar J Zico Kolter Jason D. Lee
Optimization in deep learning remains poorly understood.  A key difficulty is that optimizers exhibit complex oscillatory dynamics, referred to as "edge of stability," which cannot be captured by traditional optimization theory.  In this paper, we show that the path taken by an oscillatory optimizer can often be captured by a  _central flow_: a differential equation which directly models the time-averaged (i.e. smoothed) optimization trajectory. We empirically show that these central flows can predict long-term optimization trajectories for generic neural networks with a high degree of numerical accuracy.  By interpreting these flows, we are able to understand  how gradient descent makes progress even as the loss sometimes goes up; how adaptive optimizers ``adapt'' to the local loss landscape; and how adaptive optimizers implicitly seek out regions of weight space where they can take larger steps.  These insights (and others) are not apparent from the optimizers' update rules, but are revealed by the central flows.  Therefore, we believe that central flows constitute a promising tool for reasoning about optimization in deep learning.
Edge of Stability, Optimization Dynamics, Adaptive Optimizers, RMSProp
We show how to understand the dynamics of oscillatory optimizers in deep learning