This episode examines a Meta-led paper that develops the first systematic scaling laws for reinforcement learning in large language models, based on over 400,000 GPU-hours of experiments. The researchers propose a sigmoid framework to predict RL performance at large compute budgets from early training runs, addressing a critical gap in the field—while pre-training has well-established power-law relationships like Chinchilla scaling, RL has lacked predictive models due to shifting data distributions and bounded reward functions. The work focuses on mathematical reasoning tasks using AIME problems and introduces ScaleRL, a best-practice training recipe that successfully extrapolates performance from 50,000 to 100,000 GPU-hours. However, the hosts raise important questions about generalization beyond math to domains like code and dialogue, and whether the smooth sigmoid curves capture potential phase transitions or emergent capabilities that might appear at higher compute scales.

Sources:

1. The Art of Scaling Reinforcement Learning Compute for LLMs — Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S. Dhillon, David Brandfonbrener, Rishabh Agarwal, 2025

http://arxiv.org/abs/2510.13786v1

2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al., 2020

https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models

3. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al., 2022

https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29

4. Scaling Laws for Reward Model Overoptimization — Leo Gao, John Schulman, Jacob Hilton, 2023

https://scholar.google.com/scholar?q=Scaling+Laws+for+Reward+Model+Overoptimization

5. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI Team (Guo et al.), 2025

https://scholar.google.com/scholar?q=DeepSeek-R1%3A+Incentivizing+Reasoning+Capability+in+LLMs+via+Reinforcement+Learning

6. Proximal Policy Optimization Algorithms — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017

https://scholar.google.com/scholar?q=Proximal+Policy+Optimization+Algorithms

7. Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn, 2023

https://scholar.google.com/scholar?q=Direct+Preference+Optimization%3A+Your+Language+Model+is+Secretly+a+Reward+Model

8. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, 2022

https://scholar.google.com/scholar?q=Chain-of-Thought+Prompting+Elicits+Reasoning+in+Large+Language+Models

9. STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning — Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman, 2022

https://scholar.google.com/scholar?q=STaR%3A+Self-Taught+Reasoner+Bootstrapping+Reasoning+With+Reasoning

10. Let's Verify Step by Step — Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe, 2023

https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step

11. Reward Model Ensembles Help Mitigate Overoptimization — Thomas Coste, Usman Anwar, Robert Kirk, David Krueger, 2023

https://scholar.google.com/scholar?q=Reward+Model+Ensembles+Help+Mitigate+Overoptimization

12. DAPO: Data-Adaptive Policy Optimization — Yu et al., 2025

https://scholar.google.com/scholar?q=DAPO%3A+Data-Adaptive+Policy+Optimization

13. Chinchilla: Training Compute-Optimal Large Language Models — Hoffmann et al., 2022

https://scholar.google.com/scholar?q=Chinchilla%3A+Training+Compute-Optimal+Large+Language+Models

14. OpenAI o1 System Card — OpenAI, 2024

https://scholar.google.com/scholar?q=OpenAI+o1+System+Card

15. Direct Preference Optimization — Rafailov et al., 2023

https://scholar.google.com/scholar?q=Direct+Preference+Optimization

16. STaR: Self-Taught Reasoner — Zelikman et al., 2022

https://scholar.google.com/scholar?q=STaR%3A+Self-Taught+Reasoner

17. REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization — approximate authors unknown from snippet, recent (post-2024 based on context)

https://scholar.google.com/scholar?q=REINFORCE%2B%2B%3A+Stabilizing+Critic-Free+Policy+Optimization+with+Global+Advantage+Normalization

18. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models — approximate authors unknown from snippet, recent (likely 2024-2025)

https://scholar.google.com/scholar?q=Asynchronous+RLHF%3A+Faster+and+More+Efficient+Off-Policy+RL+for+Language+Models

19. Tapered Off-Policy REINFORCE: Stable and Efficient Reinforcement Learning for LLMs — approximate authors unknown from snippet, recent (likely 2025)

https://scholar.google.com/scholar?q=Tapered+Off-Policy+REINFORCE%3A+Stable+and+Efficient+Reinforcement+Learning+for+LLMs

20. A Survey of Post-Training Scaling in Large Language Models — approximate authors unknown from snippet, recent

https://scholar.google.com/scholar?q=A+Survey+of+Post-Training+Scaling+in+Large+Language+Models

21. Are Emergent Abilities in Large Language Models Just In-Context Learning? — approximate authors unknown from snippet, recent

https://scholar.google.com/scholar?q=Are+Emergent+Abilities+in+Large+Language+Models+Just+In-Context+Learning%3F

22. AI Post Transformers: Gradient Descent at Inference Time for LLM Reasoning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-10-gradient-descent-at-inference-time-for-l-20617d.mp3

23. AI Post Transformers: Emergent Cooperation in Self-Interested Multi-Agent AI — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-13-emergent-cooperation-in-self-interested-9c0b4c.mp3

Interactive Visualization: The Art of Scaling Reinforcement Learning Compute for LLMs