Old Optimizer, New Norm: An Anthology
Deep learning optimizers are often motivated through a mix of convex and
approximate second-order theory. We select three such methods -- Adam, Shampoo
and Prodigy -- and argue that each method can instead be understood as a
squarely first-order method without convexity assumptions. In fact, after
switching off exponential moving averages, each method is equivalent to
steepest descent under a particular norm. By generalizing this observation, we
chart a new design space for training algorithms. Different operator norms
should be assigned to different tensors based on the role that the tensor plays
within the network. For example, while linear and embedding layers may have the
same weight space of R^{mtimes n}, these layers play different
roles and should be assigned different norms. We hope that this idea of
carefully metrizing the neural architecture might lead to more stable, scalable
and indeed faster training.
66feab3f285117b16ffcc1c0
Jeremy Bernstein
66feab3f285117b16ffcc1c1
Laker Newhouse
Old Optimizer, New Norm: An Anthology
Deep learning optimizers are often motivated through a mix of convex and
approximate second-order theory. We select three such methods -- Adam, Shampoo
and Prodigy -- and argue that each method can instead be understood as a
squarely first-order method without convexity assumptions. In fact, after
switching off exponential moving averages, each method is equivalent to
steepest descent under a particular norm. By generalizing this observation, we
chart a new design space for training algorithms. Different operator norms
should be assigned to different tensors based on the role that the tensor plays
within the network. For example, while linear and embedding layers may have the
same weight space of R^{mtimes n}, these layers play different
roles and should be assigned different norms. We hope that this idea of
carefully metrizing the neural architecture might lead to more stable, scalable
and indeed faster training.
Adam, Shampoo, and Prodigy optimizers are reinterpreted as first-order methods under specific norms, suggesting a new approach to training algorithms by assigning different norms to different tensors based on their role in the network.
Adam Shampoo Prodigy first-order method exponential moving averages steepest descent operator norms neural architecture metrizing