title: Practical Efficiency of Muon for Pretraining
summary: We demonstrate that Muon, the simplest instantiation of a second-order
  optimizer, explicitly expands the Pareto frontier over AdamW on the
  compute-time tradeoff. We find that Muon is more effective than AdamW in
  retaining data efficiency at large batch sizes, far beyond the so-called
  critical batch size, while remaining computationally efficient, thus enabling
  more economical training. We study the combination of Muon and the maximal
  update parameterization (muP) for efficient hyperparameter transfer and present
  a simple telescoping algorithm that accounts for all sources of error in muP
  while introducing only a modest overhead in resources. We validate our findings
  through extensive experiments with model sizes up to four billion parameters
  and ablations on the data distribution and architecture.
paper_authors._id: 6819780dc3d212ad5b48cc07
paper_authors.name: Essential AI
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T16:53:02.976Z
paper_authors._id: 67f40efb64c87965b9063655
paper_authors.avatarUrl: /avatars/a25abd6f3318036c8d589633221c2010.svg
paper_authors.fullname: Research at Essential AI
paper_authors.type: user
paper_authors.user: Research-EAI
paper_authors._id: 6819780dc3d212ad5b48cc09
paper_authors.name: Ishaan Shah
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:33:24.884Z
paper_authors._id: 65ef97d0e5fc4abe66c05ed0
paper_authors.avatarUrl: /avatars/1d601a22639b3136bfb3519826451ddb.svg
paper_authors.fullname: Ishaan Shah
paper_authors.type: user
paper_authors.user: ishaan-essential
paper_authors._id: 6819780dc3d212ad5b48cc0a
paper_authors.name: Anthony M. Polloreno
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:01:47.445Z
paper_authors._id: 6675e3ed66c4fa6d0c10e229
paper_authors.avatarUrl: /avatars/f73c347d824a56079729c82d60d3edc3.svg
paper_authors.fullname: Anthony Polloreno
paper_authors.type: user
paper_authors.user: ampolloreno
paper_authors._id: 6819780dc3d212ad5b48cc0b
paper_authors.name: Karl Stratos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:01:53.826Z
paper_authors._id: 64d9ac38badf1110f7fcf030
paper_authors.avatarUrl: /avatars/c55c61af8dd52e6b4856684638b850a6.svg
paper_authors.fullname: Karl Stratos
paper_authors.type: user
paper_authors.user: karlstratos
paper_authors._id: 6819780dc3d212ad5b48cc0c
paper_authors.name: Philip Monk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:01:59.992Z
paper_authors._id: 66622dacec18341b268f97a6
paper_authors.avatarUrl: /avatars/8bc7d6a7c28c83aacdbeeb770716b1c0.svg
paper_authors.fullname: Philip Monk
paper_authors.type: user
paper_authors.user: monk-essential
paper_authors._id: 6819780dc3d212ad5b48cc0d
paper_authors.name: Adarsh Chaluvaraju
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:02:16.867Z
paper_authors._id: 67bfd6daca6e3c22b6de31ee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/TTt6_o9tYEgeWK26DgI_7.png
paper_authors.fullname: Adarsh Chaluvaraju
paper_authors.type: user
paper_authors.user: cadarsh-essential
paper_authors._id: 6819780dc3d212ad5b48cc0e
paper_authors.name: Andrew Hojel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:02:24.655Z
paper_authors._id: 6408e4f93461c51cf7345060
paper_authors.avatarUrl: /avatars/328b508e2de9e50dca2412adeb3542f5.svg
paper_authors.fullname: Andrew Hojel
paper_authors.type: user
paper_authors.user: andrewhojel
paper_authors._id: 6819780dc3d212ad5b48cc0f
paper_authors.name: Andrew Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:02:30.884Z
paper_authors._id: 62e24efc3a616d16e2f426ea
paper_authors.avatarUrl: /avatars/a2433c971f80e6cf738c03e843666cff.svg
paper_authors.fullname: Andrew Ma
paper_authors.type: user
paper_authors.user: AndrewMa
paper_authors._id: 6819780dc3d212ad5b48cc10
paper_authors.name: Anil Thomas
paper_authors._id: 6819780dc3d212ad5b48cc11
paper_authors.name: Ashish Tanwer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T08:59:45.682Z
paper_authors._id: 66f5bf4db839f338c5afdc9a
paper_authors.avatarUrl: /avatars/0cf00615da06cb7c04c42d55e9a97e84.svg
paper_authors.fullname: Ashish Tanwer
paper_authors.type: user
paper_authors.user: TanwerAshish
paper_authors._id: 6819780dc3d212ad5b48cc12
paper_authors.name: Darsh J Shah
paper_authors._id: 6819780dc3d212ad5b48cc13
paper_authors.name: Khoi Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:33:20.971Z
paper_authors._id: 67ed7aa9290a7f9d33113fb5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jMMTObGZktuXWJ3wVVSxj.png
paper_authors.fullname: Khoi Nguyen
paper_authors.type: user
paper_authors.user: KTLK
paper_authors._id: 6819780dc3d212ad5b48cc14
paper_authors.name: Kurt Smith
paper_authors._id: 6819780dc3d212ad5b48cc15
paper_authors.name: Michael Callahan
paper_authors._id: 6819780dc3d212ad5b48cc16
paper_authors.name: Michael Pust
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:03:14.427Z
paper_authors._id: 66cd078ea796074d428fde0f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/qThw3H6ukqIAuRy7aJTN1.jpeg
paper_authors.fullname: Michael Pust
paper_authors.type: user
paper_authors.user: essentialpust
paper_authors._id: 6819780dc3d212ad5b48cc17
paper_authors.name: Mohit Parmar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T16:52:57.811Z
paper_authors._id: 671aeee626d87414f59ee7d6
paper_authors.avatarUrl: /avatars/131d0ff4c12bc70e74945910ce0e1d56.svg
paper_authors.fullname: Mohit Parmar
paper_authors.type: user
paper_authors.user: mohit-essential
paper_authors._id: 6819780dc3d212ad5b48cc18
paper_authors.name: Peter Rushton
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:13:10.422Z
paper_authors._id: 667f4ed15555b56581ad2b78
paper_authors.avatarUrl: /avatars/18f0a166f5d4a34c4bd536655c478516.svg
paper_authors.fullname: Pete Rushton
paper_authors.type: user
paper_authors.user: prushto
paper_authors._id: 6819780dc3d212ad5b48cc19
paper_authors.name: Platon Mazarakis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:03:32.983Z
paper_authors._id: 67f4ced58c4cbc2f5d95cd17
paper_authors.avatarUrl: /avatars/3f440a59c38f5c0c7a77746ef54ed0a5.svg
paper_authors.fullname: Platon Mazarakis
paper_authors.type: user
paper_authors.user: Platona
paper_authors._id: 6819780dc3d212ad5b48cc1a
paper_authors.name: Ritvik Kapila
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:03:51.154Z
paper_authors._id: 654bdcf2e06d25def57cc54b
paper_authors.avatarUrl: /avatars/2d2612bd7072edd60876b504345fbf25.svg
paper_authors.fullname: Ritvik Kapila
paper_authors.type: user
paper_authors.user: rkapila
paper_authors._id: 6819780dc3d212ad5b48cc1b
paper_authors.name: Saurabh Srivastava
paper_authors._id: 6819780dc3d212ad5b48cc1c
paper_authors.name: Somanshu Singla
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:33:27.349Z
paper_authors._id: 679bc0b23e12a166672e5275
paper_authors.avatarUrl: /avatars/fe9d0e79c21c9d3594420e69e3809f0f.svg
paper_authors.fullname: Somanshu Singla
paper_authors.type: user
paper_authors.user: somanshu-essential
paper_authors._id: 6819780dc3d212ad5b48cc1d
paper_authors.name: Tim Romanski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:04:09.387Z
paper_authors._id: 67101a7165442ddc48cb4b07
paper_authors.avatarUrl: /avatars/551777fcd1638998ad9fd16804b313ec.svg
paper_authors.fullname: Tim Romanski
paper_authors.type: user
paper_authors.user: tim-essential
paper_authors._id: 6819780dc3d212ad5b48cc1e
paper_authors.name: Yash Vanjani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:04:16.859Z
paper_authors._id: 66f5f3f99ef08fe3c1f4c35a
paper_authors.avatarUrl: /avatars/41b8b6f90eb87e685b74587317296a1b.svg
paper_authors.fullname: Yash Vanjani
paper_authors.type: user
paper_authors.user: yash-essential
paper_authors._id: 6819780dc3d212ad5b48cc1f
paper_authors.name: Ashish Vaswani
paper_title: Practical Efficiency of Muon for Pretraining
paper_summary: We demonstrate that Muon, the simplest instantiation of a second-order
  optimizer, explicitly expands the Pareto frontier over AdamW on the
  compute-time tradeoff. We find that Muon is more effective than AdamW in
  retaining data efficiency at large batch sizes, far beyond the so-called
  critical batch size, while remaining computationally efficient, thus enabling
  more economical training. We study the combination of Muon and the maximal
  update parameterization (muP) for efficient hyperparameter transfer and present
  a simple telescoping algorithm that accounts for all sources of error in muP
  while introducing only a modest overhead in resources. We validate our findings
  through extensive experiments with model sizes up to four billion parameters
  and ablations on the data distribution and architecture.
paper_ai_summary: Muon, a second-order optimizer, improves data efficiency and computational savings over AdamW, especially at large batch sizes, and combined with muP, it provides efficient hyperparameter transfer and minimal resource overhead.
paper_ai_keywords: Muon second-order optimizer Pareto frontier AdamW data efficiency critical batch size maximal update parameterization muP hyperparameter transfer telescoping algorithm parameters data distribution architecture