Sign up to save your podcasts
Or
Share “Trust me bro, just one more RL scale up, this one will be the real scale up with the good environments, the actually legit one, trust me bro” by ryan_greenblatt
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Trust me bro, just one more RL scale up, this one will be the real scale up with the good environments, the actually legit one, trust me bro” by ryan_greenblatt
I've recently written about how I've updated against seeing substantially faster than trend AI progress due to quickly massively scaling up RL on agentic software engineering. One response I've heard is something like:
RL scale-ups so far have used very crappy environments due to difficulty quickly sourcing enough decent (or even high quality) environments. Thus, once AI companies manage to get their hands on actually good RL environments (which could happen pretty quickly), performance will increase a bunch.
Another way to put this response is that AI companies haven't actually done a good job scaling up RL—they've scaled up the compute, but with low quality data—and once they actually do the RL scale up for real this time, there will be a big jump in AI capabilities (which yields substantially above trend progress). I'm skeptical of this argument because I think that ongoing improvements to RL environments [...]
---
Outline:
(04:18) Counterargument: Actually, companies havent gotten around to improving RL environment quality until recently (or there is substantial lead time on scaling up RL environments etc.) so better RL environments didnt drive much of late 2024 and 2025 progress
(05:24) Counterargument: AIs will soon reach a critical capability threshold where AIs themselves can build high quality RL environments
(06:51) Counterargument: AI companies are massively fucking up their training runs (either pretraining or RL) and once they get their shit together more, well see fast progress
(08:34) Counterargument: This isnt that related to RL scale up, but OpenAI has some massive internal advance in verification which they demonstrated via getting IMO gold and this will cause (much) faster progress late this year or early next year
(10:12) Thoughts and speculation on scaling up the quality of RL environments
The original text contained 5 footnotes which were omitted from this narration.
---
First published:
September 3rd, 2025
Source:
https://www.lesswrong.com/posts/HsLWpZ2zad43nzvWi/trust-me-bro-just-one-more-rl-scale-up-this-one-will-be-the
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrongI've recently written about how I've updated against seeing substantially faster than trend AI progress due to quickly massively scaling up RL on agentic software engineering. One response I've heard is something like:
RL scale-ups so far have used very crappy environments due to difficulty quickly sourcing enough decent (or even high quality) environments. Thus, once AI companies manage to get their hands on actually good RL environments (which could happen pretty quickly), performance will increase a bunch.
Another way to put this response is that AI companies haven't actually done a good job scaling up RL—they've scaled up the compute, but with low quality data—and once they actually do the RL scale up for real this time, there will be a big jump in AI capabilities (which yields substantially above trend progress). I'm skeptical of this argument because I think that ongoing improvements to RL environments [...]
---
Outline:
(04:18) Counterargument: Actually, companies havent gotten around to improving RL environment quality until recently (or there is substantial lead time on scaling up RL environments etc.) so better RL environments didnt drive much of late 2024 and 2025 progress
(05:24) Counterargument: AIs will soon reach a critical capability threshold where AIs themselves can build high quality RL environments
(06:51) Counterargument: AI companies are massively fucking up their training runs (either pretraining or RL) and once they get their shit together more, well see fast progress
(08:34) Counterargument: This isnt that related to RL scale up, but OpenAI has some massive internal advance in verification which they demonstrated via getting IMO gold and this will cause (much) faster progress late this year or early next year
(10:12) Thoughts and speculation on scaling up the quality of RL environments
The original text contained 5 footnotes which were omitted from this narration.
---
First published:
September 3rd, 2025
Source:
https://www.lesswrong.com/posts/HsLWpZ2zad43nzvWi/trust-me-bro-just-one-more-rl-scale-up-this-one-will-be-the
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (30+ Karma)
View all
Making Sense with Sam Harris
26,319 Listeners
Conversations with Tyler
2,452 Listeners
The Peter Attia Drive
8,521 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,175 Listeners
ManifoldOne
93 Listeners
Your Undivided Attention
1,602 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
9,938 Listeners
Machine Learning Street Talk (MLST)
96 Listeners
Dwarkesh Podcast
517 Listeners
Hard Fork
5,509 Listeners
The Ezra Klein Show
15,892 Listeners
Moonshots with Peter Diamandis
553 Listeners
No Priors: Artificial Intelligence | Technology | Startups
131 Listeners
Latent Space: The AI Engineer Podcast
93 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
465 Listeners