title: μLO: Compute-Efficient Meta-Generalization of Learned Optimizers
summary: Learned optimizers (LOs) can significantly reduce the wall-clock training
  time of neural networks, substantially reducing training costs. However, they
  often suffer from poor meta-generalization, especially when training networks
  larger than those seen during meta-training. To address this, we use the
  recently proposed Maximal Update Parametrization (muP), which allows
  zero-shot generalization of optimizer hyperparameters from smaller to larger
  models. We extend muP theory to learned optimizers, treating the
  meta-training problem as finding the learned optimizer under muP. Our
  evaluation shows that LOs meta-trained with muP substantially improve
  meta-generalization as compared to LOs trained under standard parametrization
  (SP). Notably, when applied to large-width models, our best muLO, trained
  for 103 GPU-hours, matches or exceeds the performance of VeLO, the largest
  publicly available learned optimizer, meta-trained with 4000 TPU-months of
  compute. Moreover, muLOs demonstrate better generalization than their SP
  counterparts to deeper networks and to much longer training horizons (25 times
  longer) than those seen during meta-training.
paper_authors._id: 665e929101ca1c80a0cef855
paper_authors.name: Benjamin Thérien
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:38:07.719Z
paper_authors._id: 62d0e2a3fb896639b2a04ca1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d0e2a3fb896639b2a04ca1/EE4gRL6XSUi32rFiTh4S1.jpeg
paper_authors.fullname: Benjamin Therien
paper_authors.type: user
paper_authors.user: btherien
paper_authors._id: 665e929101ca1c80a0cef856
paper_authors.name: Charles-Étienne Joseph
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:38:13.933Z
paper_authors._id: 65c0016fdaa6f48256f0799f
paper_authors.avatarUrl: /avatars/93658acaaaee76eca64df6b269172cc6.svg
paper_authors.fullname: Charles-Étienne Joseph
paper_authors.type: user
paper_authors.user: lefameuxbeding
paper_authors._id: 665e929101ca1c80a0cef857
paper_authors.name: Boris Knyazev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:38:19.906Z
paper_authors._id: 63e16165f0039731dfdd442a
paper_authors.avatarUrl: /avatars/37cf99dc016c291c800f60d260173482.svg
paper_authors.fullname: Boris Knyazev
paper_authors.type: user
paper_authors.user: bknyaz
paper_authors._id: 665e929101ca1c80a0cef858
paper_authors.name: Edouard Oyallon
paper_authors._id: 665e929101ca1c80a0cef859
paper_authors.name: Irina Rish
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:38:35.376Z
paper_authors._id: 6227f764236b7b2eefa7f173
paper_authors.avatarUrl: /avatars/0204e3da8e9ef12d88c98c6e955f8340.svg
paper_authors.fullname: Irina Rish
paper_authors.type: user
paper_authors.user: irinarish
paper_authors._id: 665e929101ca1c80a0cef85a
paper_authors.name: Eugene Belilovsky
paper_title: μLO: Compute-Efficient Meta-Generalization of Learned Optimizers
paper_summary: Learned optimizers (LOs) can significantly reduce the wall-clock training
  time of neural networks, substantially reducing training costs. However, they
  often suffer from poor meta-generalization, especially when training networks
  larger than those seen during meta-training. To address this, we use the
  recently proposed Maximal Update Parametrization (muP), which allows
  zero-shot generalization of optimizer hyperparameters from smaller to larger
  models. We extend muP theory to learned optimizers, treating the
  meta-training problem as finding the learned optimizer under muP. Our
  evaluation shows that LOs meta-trained with muP substantially improve
  meta-generalization as compared to LOs trained under standard parametrization
  (SP). Notably, when applied to large-width models, our best muLO, trained
  for 103 GPU-hours, matches or exceeds the performance of VeLO, the largest
  publicly available learned optimizer, meta-trained with 4000 TPU-months of
  compute. Moreover, muLOs demonstrate better generalization than their SP
  counterparts to deeper networks and to much longer training horizons (25 times
  longer) than those seen during meta-training.
paper_ai_keywords: Learned optimizers (LOs) Maximal Update Parametrization ($\mu$P) zero-shot generalization optimizer hyperparameters meta-training problem standard parametrization (SP) VeLO TPUs GPU-hours meta-generalization memory usage training horizons