Promoting Exploration in Memory-Augmented Adam using Critical Momenta
Adaptive gradient-based optimizers, particularly Adam, have left their mark
in training large-scale deep learning models. The strength of such optimizers
is that they exhibit fast convergence while being more robust to hyperparameter
choice. However, they often generalize worse than non-adaptive methods. Recent
studies have tied this performance gap to flat minima selection: adaptive
methods tend to find solutions in sharper basins of the loss landscape, which
in turn hurts generalization. To overcome this issue, we propose a new
memory-augmented version of Adam that promotes exploration towards flatter
minima by using a buffer of critical momentum terms during training.
Intuitively, the use of the buffer makes the optimizer overshoot outside the
basin of attraction if it is not wide enough. We empirically show that our
method improves the performance of several variants of Adam on standard
supervised language modelling and image classification tasks.
64b897f0f1f8e6ea5853a597
Pranshu Malviya
claimed_verified
2023-07-20T14:13:55.366Z
64b93f0d91876a8d16d470ab
/avatars/589ffce1eed8f28a9e1a001f36e0838e.svg
Pranshu Malviya
user
pranshu28
64b897f0f1f8e6ea5853a598
Gonçalo Mordido
64b897f0f1f8e6ea5853a599
Aristide Baratin
64b897f0f1f8e6ea5853a59a
Reza Babanezhad Harikandeh
64b897f0f1f8e6ea5853a59b
Jerry Huang
claimed_verified
2023-07-21T07:36:28.759Z
64b94c1635c815492d40c509
/avatars/cd2160754dc89b440ca81d30db030fe3.svg
Jerry Huang
user
jhuang265
64b897f0f1f8e6ea5853a59c
Simon Lacoste-Julien
admin_assigned
2023-07-20T09:46:42.691Z
6406bcaca577649430c6bff4
/avatars/77b386172b0d86d40cddd7a7a8744491.svg
Simon Lacoste-Julien
user
slacoste
64b897f0f1f8e6ea5853a59d
Razvan Pascanu
claimed_verified
2023-07-20T14:13:59.202Z
64b9310403124195cd9778ec
/avatars/57c594d3d0f97d3010b15b6a0806451c.svg
Razvan Pascanu
user
razp
64b897f0f1f8e6ea5853a59e
Sarath Chandar
Promoting Exploration in Memory-Augmented Adam using Critical Momenta
Adaptive gradient-based optimizers, particularly Adam, have left their mark
in training large-scale deep learning models. The strength of such optimizers
is that they exhibit fast convergence while being more robust to hyperparameter
choice. However, they often generalize worse than non-adaptive methods. Recent
studies have tied this performance gap to flat minima selection: adaptive
methods tend to find solutions in sharper basins of the loss landscape, which
in turn hurts generalization. To overcome this issue, we propose a new
memory-augmented version of Adam that promotes exploration towards flatter
minima by using a buffer of critical momentum terms during training.
Intuitively, the use of the buffer makes the optimizer overshoot outside the
basin of attraction if it is not wide enough. We empirically show that our
method improves the performance of several variants of Adam on standard
supervised language modelling and image classification tasks.
A memory-augmented version of Adam optimizer enhances generalization by encouraging exploration towards flatter minima in the loss landscape.
gradient-based optimizers Adam fast convergence hyperparameter choice flat minima sharp basins loss landscape momentum terms buffer exploration overhooting supervised language modelling image classification