Practical Efficiency of Muon for Pretraining
We demonstrate that Muon, the simplest instantiation of a second-order
optimizer, explicitly expands the Pareto frontier over AdamW on the
compute-time tradeoff. We find that Muon is more effective than AdamW in
retaining data efficiency at large batch sizes, far beyond the so-called
critical batch size, while remaining computationally efficient, thus enabling
more economical training. We study the combination of Muon and the maximal
update parameterization (muP) for efficient hyperparameter transfer and present
a simple telescoping algorithm that accounts for all sources of error in muP
while introducing only a modest overhead in resources. We validate our findings
through extensive experiments with model sizes up to four billion parameters
and ablations on the data distribution and architecture.
6819780dc3d212ad5b48cc07
Essential AI
claimed_verified
2025-05-06T16:53:02.976Z
67f40efb64c87965b9063655
/avatars/a25abd6f3318036c8d589633221c2010.svg
Research at Essential AI
user
Research-EAI
6819780dc3d212ad5b48cc09
Ishaan Shah
claimed_verified
2025-05-06T08:33:24.884Z
65ef97d0e5fc4abe66c05ed0
/avatars/1d601a22639b3136bfb3519826451ddb.svg
Ishaan Shah
user
ishaan-essential
6819780dc3d212ad5b48cc0a
Anthony M. Polloreno
admin_assigned
2025-05-06T09:01:47.445Z
6675e3ed66c4fa6d0c10e229
/avatars/f73c347d824a56079729c82d60d3edc3.svg
Anthony Polloreno
user
ampolloreno
6819780dc3d212ad5b48cc0b
Karl Stratos
admin_assigned
2025-05-06T09:01:53.826Z
64d9ac38badf1110f7fcf030
/avatars/c55c61af8dd52e6b4856684638b850a6.svg
Karl Stratos
user
karlstratos
6819780dc3d212ad5b48cc0c
Philip Monk
admin_assigned
2025-05-06T09:01:59.992Z
66622dacec18341b268f97a6
/avatars/8bc7d6a7c28c83aacdbeeb770716b1c0.svg
Philip Monk
user
monk-essential
6819780dc3d212ad5b48cc0d
Adarsh Chaluvaraju
admin_assigned
2025-05-06T09:02:16.867Z
67bfd6daca6e3c22b6de31ee
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/TTt6_o9tYEgeWK26DgI_7.png
Adarsh Chaluvaraju
user
cadarsh-essential
6819780dc3d212ad5b48cc0e
Andrew Hojel
admin_assigned
2025-05-06T09:02:24.655Z
6408e4f93461c51cf7345060
/avatars/328b508e2de9e50dca2412adeb3542f5.svg
Andrew Hojel
user
andrewhojel
6819780dc3d212ad5b48cc0f
Andrew Ma
admin_assigned
2025-05-06T09:02:30.884Z
62e24efc3a616d16e2f426ea
/avatars/a2433c971f80e6cf738c03e843666cff.svg
Andrew Ma
user
AndrewMa
6819780dc3d212ad5b48cc10
Anil Thomas
6819780dc3d212ad5b48cc11
Ashish Tanwer
claimed_verified
2025-05-08T08:59:45.682Z
66f5bf4db839f338c5afdc9a
/avatars/0cf00615da06cb7c04c42d55e9a97e84.svg
Ashish Tanwer
user
TanwerAshish
6819780dc3d212ad5b48cc12
Darsh J Shah
6819780dc3d212ad5b48cc13
Khoi Nguyen
claimed_verified
2025-05-06T08:33:20.971Z
67ed7aa9290a7f9d33113fb5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jMMTObGZktuXWJ3wVVSxj.png
Khoi Nguyen
user
KTLK
6819780dc3d212ad5b48cc14
Kurt Smith
6819780dc3d212ad5b48cc15
Michael Callahan
claimed_verified
2025-06-23T08:32:56.751Z
66f62e1aa854f31fea6c7d25
/avatars/3cf6f471e26cec430fa5623bac938fa7.svg
Michael Callahan
user
mikecallahan
6819780dc3d212ad5b48cc16
Michael Pust
admin_assigned
2025-05-06T09:03:14.427Z
66cd078ea796074d428fde0f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/qThw3H6ukqIAuRy7aJTN1.jpeg
Michael Pust
user
essentialpust
6819780dc3d212ad5b48cc17
Mohit Parmar
claimed_verified
2025-05-06T16:52:57.811Z
671aeee626d87414f59ee7d6
/avatars/131d0ff4c12bc70e74945910ce0e1d56.svg
Mohit Parmar
user
mohit-essential
6819780dc3d212ad5b48cc18
Peter Rushton
claimed_verified
2025-05-13T08:13:10.422Z
667f4ed15555b56581ad2b78
/avatars/18f0a166f5d4a34c4bd536655c478516.svg
Pete Rushton
user
prushto
6819780dc3d212ad5b48cc19
Platon Mazarakis
admin_assigned
2025-05-06T09:03:32.983Z
67f4ced58c4cbc2f5d95cd17
/avatars/3f440a59c38f5c0c7a77746ef54ed0a5.svg
Platon Mazarakis
user
Platona
6819780dc3d212ad5b48cc1a
Ritvik Kapila
admin_assigned
2025-05-06T09:03:51.154Z
654bdcf2e06d25def57cc54b
/avatars/2d2612bd7072edd60876b504345fbf25.svg
Ritvik Kapila
user
rkapila
6819780dc3d212ad5b48cc1b
Saurabh Srivastava
6819780dc3d212ad5b48cc1c
Somanshu Singla
claimed_verified
2025-05-06T08:33:27.349Z
679bc0b23e12a166672e5275
/avatars/fe9d0e79c21c9d3594420e69e3809f0f.svg
Somanshu Singla
user
SomanshuSingla
6819780dc3d212ad5b48cc1d
Tim Romanski
admin_assigned
2025-05-06T09:04:09.387Z
67101a7165442ddc48cb4b07
/avatars/551777fcd1638998ad9fd16804b313ec.svg
Tim Romanski
user
tim-essential
6819780dc3d212ad5b48cc1e
Yash Vanjani
admin_assigned
2025-05-06T09:04:16.859Z
66f5f3f99ef08fe3c1f4c35a
/avatars/41b8b6f90eb87e685b74587317296a1b.svg
Yash Vanjani
user
yash-essential
6819780dc3d212ad5b48cc1f
Ashish Vaswani
Practical Efficiency of Muon for Pretraining
We demonstrate that Muon, the simplest instantiation of a second-order
optimizer, explicitly expands the Pareto frontier over AdamW on the
compute-time tradeoff. We find that Muon is more effective than AdamW in
retaining data efficiency at large batch sizes, far beyond the so-called
critical batch size, while remaining computationally efficient, thus enabling
more economical training. We study the combination of Muon and the maximal
update parameterization (muP) for efficient hyperparameter transfer and present
a simple telescoping algorithm that accounts for all sources of error in muP
while introducing only a modest overhead in resources. We validate our findings
through extensive experiments with model sizes up to four billion parameters
and ablations on the data distribution and architecture.
Muon, a second-order optimizer, improves data efficiency and computational savings over AdamW, especially at large batch sizes, and combined with muP, it provides efficient hyperparameter transfer and minimal resource overhead.
Muon second-order optimizer Pareto frontier AdamW data efficiency critical batch size maximal update parameterization muP hyperparameter transfer telescoping algorithm parameters data distribution architecture