title: Promoting Exploration in Memory-Augmented Adam using Critical Momenta
summary: Adaptive gradient-based optimizers, particularly Adam, have left their mark
  in training large-scale deep learning models. The strength of such optimizers
  is that they exhibit fast convergence while being more robust to hyperparameter
  choice. However, they often generalize worse than non-adaptive methods. Recent
  studies have tied this performance gap to flat minima selection: adaptive
  methods tend to find solutions in sharper basins of the loss landscape, which
  in turn hurts generalization. To overcome this issue, we propose a new
  memory-augmented version of Adam that promotes exploration towards flatter
  minima by using a buffer of critical momentum terms during training.
  Intuitively, the use of the buffer makes the optimizer overshoot outside the
  basin of attraction if it is not wide enough. We empirically show that our
  method improves the performance of several variants of Adam on standard
  supervised language modelling and image classification tasks.
paper_authors._id: 64b897f0f1f8e6ea5853a597
paper_authors.name: Pranshu Malviya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-20T14:13:55.366Z
paper_authors._id: 64b93f0d91876a8d16d470ab
paper_authors.avatarUrl: /avatars/589ffce1eed8f28a9e1a001f36e0838e.svg
paper_authors.fullname: Pranshu Malviya
paper_authors.type: user
paper_authors.user: pranshu28
paper_authors._id: 64b897f0f1f8e6ea5853a598
paper_authors.name: Gonçalo Mordido
paper_authors._id: 64b897f0f1f8e6ea5853a599
paper_authors.name: Aristide Baratin
paper_authors._id: 64b897f0f1f8e6ea5853a59a
paper_authors.name: Reza Babanezhad Harikandeh
paper_authors._id: 64b897f0f1f8e6ea5853a59b
paper_authors.name: Jerry Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-21T07:36:28.759Z
paper_authors._id: 64b94c1635c815492d40c509
paper_authors.avatarUrl: /avatars/cd2160754dc89b440ca81d30db030fe3.svg
paper_authors.fullname: Jerry Huang
paper_authors.type: user
paper_authors.user: jhuang265
paper_authors._id: 64b897f0f1f8e6ea5853a59c
paper_authors.name: Simon Lacoste-Julien
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-20T09:46:42.691Z
paper_authors._id: 6406bcaca577649430c6bff4
paper_authors.avatarUrl: /avatars/77b386172b0d86d40cddd7a7a8744491.svg
paper_authors.fullname: Simon Lacoste-Julien
paper_authors.type: user
paper_authors.user: slacoste
paper_authors._id: 64b897f0f1f8e6ea5853a59d
paper_authors.name: Razvan Pascanu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-20T14:13:59.202Z
paper_authors._id: 64b9310403124195cd9778ec
paper_authors.avatarUrl: /avatars/57c594d3d0f97d3010b15b6a0806451c.svg
paper_authors.fullname: Razvan Pascanu
paper_authors.type: user
paper_authors.user: razp
paper_authors._id: 64b897f0f1f8e6ea5853a59e
paper_authors.name: Sarath Chandar
paper_title: Promoting Exploration in Memory-Augmented Adam using Critical Momenta
paper_summary: Adaptive gradient-based optimizers, particularly Adam, have left their mark
  in training large-scale deep learning models. The strength of such optimizers
  is that they exhibit fast convergence while being more robust to hyperparameter
  choice. However, they often generalize worse than non-adaptive methods. Recent
  studies have tied this performance gap to flat minima selection: adaptive
  methods tend to find solutions in sharper basins of the loss landscape, which
  in turn hurts generalization. To overcome this issue, we propose a new
  memory-augmented version of Adam that promotes exploration towards flatter
  minima by using a buffer of critical momentum terms during training.
  Intuitively, the use of the buffer makes the optimizer overshoot outside the
  basin of attraction if it is not wide enough. We empirically show that our
  method improves the performance of several variants of Adam on standard
  supervised language modelling and image classification tasks.
paper_ai_keywords: Adam adaptive gradient-based optimizers hyperparameter fast convergencegence generalization flat minima loss landscape basins memory-augmented momentum distribution communication performance supervised language modelling classification tasks