title: Old Optimizer, New Norm: An Anthology
summary: Deep learning optimizers are often motivated through a mix of convex and
  approximate second-order theory. We select three such methods -- Adam, Shampoo
  and Prodigy -- and argue that each method can instead be understood as a
  squarely first-order method without convexity assumptions. In fact, after
  switching off exponential moving averages, each method is equivalent to
  steepest descent under a particular norm. By generalizing this observation, we
  chart a new design space for training algorithms. Different operator norms
  should be assigned to different tensors based on the role that the tensor plays
  within the network. For example, while linear and embedding layers may have the
  same weight space of R^{mtimes n}, these layers play different
  roles and should be assigned different norms. We hope that this idea of
  carefully metrizing the neural architecture might lead to more stable, scalable
  and indeed faster training.
paper_authors._id: 66feab3f285117b16ffcc1c0
paper_authors.name: Jeremy Bernstein
paper_authors._id: 66feab3f285117b16ffcc1c1
paper_authors.name: Laker Newhouse
paper_title: Old Optimizer, New Norm: An Anthology
paper_summary: Deep learning optimizers are often motivated through a mix of convex and
  approximate second-order theory. We select three such methods -- Adam, Shampoo
  and Prodigy -- and argue that each method can instead be understood as a
  squarely first-order method without convexity assumptions. In fact, after
  switching off exponential moving averages, each method is equivalent to
  steepest descent under a particular norm. By generalizing this observation, we
  chart a new design space for training algorithms. Different operator norms
  should be assigned to different tensors based on the role that the tensor plays
  within the network. For example, while linear and embedding layers may have the
  same weight space of R^{mtimes n}, these layers play different
  roles and should be assigned different norms. We hope that this idea of
  carefully metrizing the neural architecture might lead to more stable, scalable
  and indeed faster training.
paper_ai_summary: Adam, Shampoo, and Prodigy optimizers are reinterpreted as first-order methods under specific norms, suggesting a new approach to training algorithms by assigning different norms to different tensors based on their role in the network.
paper_ai_keywords: Adam Shampoo Prodigy first-order method exponential moving averages steepest descent operator norms neural architecture metrizing