Sign up to save your podcasts
Or
Share Muon Is Scalable for LLM Training
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Muon Is Scalable for LLM Training
This episode explores whether the Muon optimizer can truly scale from promising small experiments to frontier-style language model training, and what it would mean if its reported roughly 2x compute-efficiency gain over AdamW holds up. It examines how Muon differs from standard optimizer setups by applying momentum plus orthogonalization to matrix-shaped hidden-layer weights while keeping embeddings and one-dimensional parameters on AdamW, making the method a deliberately hybrid training recipe rather than a pure drop-in replacement. The discussion digs into the paper’s central technical claims around scaling laws, optimizer attribution, and systems implementation, with particular attention to evidence that added weight decay and per-parameter update scaling are essential for long-run stability and preventing runaway magnitudes in bf16 training. A listener would find it interesting because the conversation goes beyond headline gains to ask whether Muon really shifts the compute-optimal frontier for LLM training or whether the result depends on a carefully engineered package whose practical value lies in how all the pieces work together.
Sources:
1. Muon is Scalable for LLM Training — Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, Zhilin Yang, 2025
http://arxiv.org/abs/2502.16982
2. Muon: An optimizer for hidden layers in neural networks — Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, Jeremy Bernstein, 2024
https://scholar.google.com/scholar?q=Muon%3A+An+optimizer+for+hidden+layers+in+neural+networks
3. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford and others, 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models
4. Scalable Second Order Optimization for Deep Learning — Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, Yoram Singer, 2020
https://scholar.google.com/scholar?q=Scalable+Second+Order+Optimization+for+Deep+Learning
5. A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks at-Scale — Hao-Jun Michael Shi, Erik M. Nitanda, Animesh Garg, Rohan Anil and others, 2023
https://scholar.google.com/scholar?q=A+Distributed+Data-Parallel+PyTorch+Implementation+of+the+Distributed+Shampoo+Optimizer+for+Training+Neural+Networks+at-Scale
6. Modular Duality in Deep Learning — Jeremy Bernstein, Laker Newhouse and collaborators, 2024
https://scholar.google.com/scholar?q=Modular+Duality+in+Deep+Learning
7. Adam-mini: Use Fewer Learning Rates To Gain More — Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, Ruoyu Sun, 2024
https://scholar.google.com/scholar?q=Adam-mini%3A+Use+Fewer+Learning+Rates+To+Gain+More
8. Small-scale proxies for large-scale transformer training instabilities — approx. recent transformer-scaling/optimization authors, recent
https://scholar.google.com/scholar?q=Small-scale+proxies+for+large-scale+transformer+training+instabilities
9. An Empirical Study of μP Learning Rate Transfer — approx. μP / maximal-update-parameterization authors, recent
https://scholar.google.com/scholar?q=An+Empirical+Study+of+%CE%BCP+Learning+Rate+Transfer
10. DeepNet: Scaling Transformers to 1,000 Layers — approx. DeepNet authors, 2022
https://scholar.google.com/scholar?q=DeepNet%3A+Scaling+Transformers+to+1%2C000+Layers
11. Why Do We Need Weight Decay in Modern Deep Learning? — approx. recent optimization authors, recent
https://scholar.google.com/scholar?q=Why+Do+We+Need+Weight+Decay+in+Modern+Deep+Learning%3F
12. Orthogonal Subspace Learning for Language Model Continual Learning — approx. continual-learning / language-model authors, recent
https://scholar.google.com/scholar?q=Orthogonal+Subspace+Learning+for+Language+Model+Continual+Learning
13. AI Post Transformers: When Spectral Gradient Updates Help Deep Learning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-when-spectral-gradient-updates-help-deep-9c8441.mp3
Interactive Visualization: Muon Is Scalable for LLM Training
View all episodes
By mcgrofThis episode explores whether the Muon optimizer can truly scale from promising small experiments to frontier-style language model training, and what it would mean if its reported roughly 2x compute-efficiency gain over AdamW holds up. It examines how Muon differs from standard optimizer setups by applying momentum plus orthogonalization to matrix-shaped hidden-layer weights while keeping embeddings and one-dimensional parameters on AdamW, making the method a deliberately hybrid training recipe rather than a pure drop-in replacement. The discussion digs into the paper’s central technical claims around scaling laws, optimizer attribution, and systems implementation, with particular attention to evidence that added weight decay and per-parameter update scaling are essential for long-run stability and preventing runaway magnitudes in bf16 training. A listener would find it interesting because the conversation goes beyond headline gains to ask whether Muon really shifts the compute-optimal frontier for LLM training or whether the result depends on a carefully engineered package whose practical value lies in how all the pieces work together.
Sources:
1. Muon is Scalable for LLM Training — Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, Zhilin Yang, 2025
http://arxiv.org/abs/2502.16982
2. Muon: An optimizer for hidden layers in neural networks — Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, Jeremy Bernstein, 2024
https://scholar.google.com/scholar?q=Muon%3A+An+optimizer+for+hidden+layers+in+neural+networks
3. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford and others, 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models
4. Scalable Second Order Optimization for Deep Learning — Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, Yoram Singer, 2020
https://scholar.google.com/scholar?q=Scalable+Second+Order+Optimization+for+Deep+Learning
5. A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks at-Scale — Hao-Jun Michael Shi, Erik M. Nitanda, Animesh Garg, Rohan Anil and others, 2023
https://scholar.google.com/scholar?q=A+Distributed+Data-Parallel+PyTorch+Implementation+of+the+Distributed+Shampoo+Optimizer+for+Training+Neural+Networks+at-Scale
6. Modular Duality in Deep Learning — Jeremy Bernstein, Laker Newhouse and collaborators, 2024
https://scholar.google.com/scholar?q=Modular+Duality+in+Deep+Learning
7. Adam-mini: Use Fewer Learning Rates To Gain More — Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, Ruoyu Sun, 2024
https://scholar.google.com/scholar?q=Adam-mini%3A+Use+Fewer+Learning+Rates+To+Gain+More
8. Small-scale proxies for large-scale transformer training instabilities — approx. recent transformer-scaling/optimization authors, recent
https://scholar.google.com/scholar?q=Small-scale+proxies+for+large-scale+transformer+training+instabilities
9. An Empirical Study of μP Learning Rate Transfer — approx. μP / maximal-update-parameterization authors, recent
https://scholar.google.com/scholar?q=An+Empirical+Study+of+%CE%BCP+Learning+Rate+Transfer
10. DeepNet: Scaling Transformers to 1,000 Layers — approx. DeepNet authors, 2022
https://scholar.google.com/scholar?q=DeepNet%3A+Scaling+Transformers+to+1%2C000+Layers
11. Why Do We Need Weight Decay in Modern Deep Learning? — approx. recent optimization authors, recent
https://scholar.google.com/scholar?q=Why+Do+We+Need+Weight+Decay+in+Modern+Deep+Learning%3F
12. Orthogonal Subspace Learning for Language Model Continual Learning — approx. continual-learning / language-model authors, recent
https://scholar.google.com/scholar?q=Orthogonal+Subspace+Learning+for+Language+Model+Continual+Learning
13. AI Post Transformers: When Spectral Gradient Updates Help Deep Learning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-when-spectral-gradient-updates-help-deep-9c8441.mp3
Interactive Visualization: Muon Is Scalable for LLM Training