Sign up to save your podcasts
Or
Share “Trust me bro, just one more RL scale up, this one will be the real scale up with the good environments, the actually legit one, trust me bro” by ryan_greenblatt
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Trust me bro, just one more RL scale up, this one will be the real scale up with the good environments, the actually legit one, trust me bro” by ryan_greenblatt
I've recently written about how I've updated against seeing substantially faster than trend AI progress due to quickly massively scaling up RL on agentic software engineering. One response I've heard is something like:
RL scale-ups so far have used very crappy environments due to difficulty quickly sourcing enough decent (or even high quality) environments. Thus, once AI companies manage to get their hands on actually good RL environments (which could happen pretty quickly), performance will increase a bunch.
Another way to put this response is that AI companies haven't actually done a good job scaling up RL—they've scaled up the compute, but with low quality data—and once they actually do the RL scale up for real this time, there will be a big jump in AI capabilities (which yields substantially above trend progress). I'm skeptical of this argument because I think that ongoing improvements to RL environments [...]
---
Outline:
(04:18) Counterargument: Actually, companies havent gotten around to improving RL environment quality until recently (or there is substantial lead time on scaling up RL environments etc.) so better RL environments didnt drive much of late 2024 and 2025 progress
(05:24) Counterargument: AIs will soon reach a critical capability threshold where AIs themselves can build high quality RL environments
(06:51) Counterargument: AI companies are massively fucking up their training runs (either pretraining or RL) and once they get their shit together more, well see fast progress
(08:34) Counterargument: This isnt that related to RL scale up, but OpenAI has some massive internal advance in verification which they demonstrated via getting IMO gold and this will cause (much) faster progress late this year or early next year
(10:12) Thoughts and speculation on scaling up the quality of RL environments
The original text contained 5 footnotes which were omitted from this narration.
---
First published:
September 3rd, 2025
Source:
https://www.lesswrong.com/posts/HsLWpZ2zad43nzvWi/trust-me-bro-just-one-more-rl-scale-up-this-one-will-be-the
---
Narrated by TYPE III AUDIO.
RL scale-ups so far have used very crappy environments due to difficulty quickly sourcing enough decent (or even high quality) environments. Thus, once AI companies manage to get their hands on actually good RL environments (which could happen pretty quickly), performance will increase a bunch.
Another way to put this response is that AI companies haven't actually done a good job scaling up RL—they've scaled up the compute, but with low quality data—and once they actually do the RL scale up for real this time, there will be a big jump in AI capabilities (which yields substantially above trend progress). I'm skeptical of this argument because I think that ongoing improvements to RL environments [...]
---
Outline:
(04:18) Counterargument: Actually, companies havent gotten around to improving RL environment quality until recently (or there is substantial lead time on scaling up RL environments etc.) so better RL environments didnt drive much of late 2024 and 2025 progress
(05:24) Counterargument: AIs will soon reach a critical capability threshold where AIs themselves can build high quality RL environments
(06:51) Counterargument: AI companies are massively fucking up their training runs (either pretraining or RL) and once they get their shit together more, well see fast progress
(08:34) Counterargument: This isnt that related to RL scale up, but OpenAI has some massive internal advance in verification which they demonstrated via getting IMO gold and this will cause (much) faster progress late this year or early next year
(10:12) Thoughts and speculation on scaling up the quality of RL environments
The original text contained 5 footnotes which were omitted from this narration.
---
First published:
September 3rd, 2025
Source:
https://www.lesswrong.com/posts/HsLWpZ2zad43nzvWi/trust-me-bro-just-one-more-rl-scale-up-this-one-will-be-the
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrong
4.8
1212 ratings
I've recently written about how I've updated against seeing substantially faster than trend AI progress due to quickly massively scaling up RL on agentic software engineering. One response I've heard is something like:
RL scale-ups so far have used very crappy environments due to difficulty quickly sourcing enough decent (or even high quality) environments. Thus, once AI companies manage to get their hands on actually good RL environments (which could happen pretty quickly), performance will increase a bunch.
Another way to put this response is that AI companies haven't actually done a good job scaling up RL—they've scaled up the compute, but with low quality data—and once they actually do the RL scale up for real this time, there will be a big jump in AI capabilities (which yields substantially above trend progress). I'm skeptical of this argument because I think that ongoing improvements to RL environments [...]
---
Outline:
(04:18) Counterargument: Actually, companies havent gotten around to improving RL environment quality until recently (or there is substantial lead time on scaling up RL environments etc.) so better RL environments didnt drive much of late 2024 and 2025 progress
(05:24) Counterargument: AIs will soon reach a critical capability threshold where AIs themselves can build high quality RL environments
(06:51) Counterargument: AI companies are massively fucking up their training runs (either pretraining or RL) and once they get their shit together more, well see fast progress
(08:34) Counterargument: This isnt that related to RL scale up, but OpenAI has some massive internal advance in verification which they demonstrated via getting IMO gold and this will cause (much) faster progress late this year or early next year
(10:12) Thoughts and speculation on scaling up the quality of RL environments
The original text contained 5 footnotes which were omitted from this narration.
---
First published:
September 3rd, 2025
Source:
https://www.lesswrong.com/posts/HsLWpZ2zad43nzvWi/trust-me-bro-just-one-more-rl-scale-up-this-one-will-be-the
---
Narrated by TYPE III AUDIO.
RL scale-ups so far have used very crappy environments due to difficulty quickly sourcing enough decent (or even high quality) environments. Thus, once AI companies manage to get their hands on actually good RL environments (which could happen pretty quickly), performance will increase a bunch.
Another way to put this response is that AI companies haven't actually done a good job scaling up RL—they've scaled up the compute, but with low quality data—and once they actually do the RL scale up for real this time, there will be a big jump in AI capabilities (which yields substantially above trend progress). I'm skeptical of this argument because I think that ongoing improvements to RL environments [...]
---
Outline:
(04:18) Counterargument: Actually, companies havent gotten around to improving RL environment quality until recently (or there is substantial lead time on scaling up RL environments etc.) so better RL environments didnt drive much of late 2024 and 2025 progress
(05:24) Counterargument: AIs will soon reach a critical capability threshold where AIs themselves can build high quality RL environments
(06:51) Counterargument: AI companies are massively fucking up their training runs (either pretraining or RL) and once they get their shit together more, well see fast progress
(08:34) Counterargument: This isnt that related to RL scale up, but OpenAI has some massive internal advance in verification which they demonstrated via getting IMO gold and this will cause (much) faster progress late this year or early next year
(10:12) Thoughts and speculation on scaling up the quality of RL environments
The original text contained 5 footnotes which were omitted from this narration.
---
First published:
September 3rd, 2025
Source:
https://www.lesswrong.com/posts/HsLWpZ2zad43nzvWi/trust-me-bro-just-one-more-rl-scale-up-this-one-will-be-the
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (Curated & Popular)
View all
Making Sense with Sam Harris
26,371 Listeners
Conversations with Tyler
2,426 Listeners
Robert Wright's Nonzero
590 Listeners
Future of Life Institute Podcast
107 Listeners
The Good Fight
905 Listeners
ManifoldOne
92 Listeners
The Prof G Pod with Scott Galloway
5,483 Listeners
Machine Learning Street Talk (MLST)
89 Listeners
Dwarkesh Podcast
489 Listeners
Hard Fork
5,473 Listeners
Clearer Thinking with Spencer Greenberg
132 Listeners
Complex Systems with Patrick McKenzie (patio11)
133 Listeners
The Marginal Revolution Podcast
93 Listeners
Statecraft
34 Listeners
The Last Invention
292 Listeners