Sign up to save your podcasts
Or
Share “Instrumental Goals Are A Different And Friendlier Kind Of Thing Than Terminal Goals” by johnswentworth, David Lorell
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Instrumental Goals Are A Different And Friendlier Kind Of Thing Than Terminal Goals” by johnswentworth, David Lorell
The Cake
Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that's fine. Baking that cake is my terminal goal.
In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.
My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile [...]
---
Outline:
(00:07) The Cake
(01:50) The Restaurant
(03:50) Happy Instrumental Convergence?
(06:27) All The Way Up
(08:05) Research Threads
---
First published:
January 24th, 2025
Source:
https://www.lesswrong.com/posts/7Z4WC4AFgfmZ3fCDC/instrumental-goals-are-a-different-and-friendlier-kind-of
---
Narrated by TYPE III AUDIO.
Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that's fine. Baking that cake is my terminal goal.
In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.
My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile [...]
---
Outline:
(00:07) The Cake
(01:50) The Restaurant
(03:50) Happy Instrumental Convergence?
(06:27) All The Way Up
(08:05) Research Threads
---
First published:
January 24th, 2025
Source:
https://www.lesswrong.com/posts/7Z4WC4AFgfmZ3fCDC/instrumental-goals-are-a-different-and-friendlier-kind-of
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrong
4.8
1111 ratings
The Cake
Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that's fine. Baking that cake is my terminal goal.
In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.
My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile [...]
---
Outline:
(00:07) The Cake
(01:50) The Restaurant
(03:50) Happy Instrumental Convergence?
(06:27) All The Way Up
(08:05) Research Threads
---
First published:
January 24th, 2025
Source:
https://www.lesswrong.com/posts/7Z4WC4AFgfmZ3fCDC/instrumental-goals-are-a-different-and-friendlier-kind-of
---
Narrated by TYPE III AUDIO.
Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that's fine. Baking that cake is my terminal goal.
In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.
My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile [...]
---
Outline:
(00:07) The Cake
(01:50) The Restaurant
(03:50) Happy Instrumental Convergence?
(06:27) All The Way Up
(08:05) Research Threads
---
First published:
January 24th, 2025
Source:
https://www.lesswrong.com/posts/7Z4WC4AFgfmZ3fCDC/instrumental-goals-are-a-different-and-friendlier-kind-of
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (Curated & Popular)
View all
Conversations with Tyler
2,382 Listeners
Astral Codex Ten Podcast
122 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,079 Listeners
ManifoldOne
87 Listeners
The Jim Rutt Show
249 Listeners
Machine Learning Street Talk (MLST)
90 Listeners
Dwarkesh Podcast
324 Listeners
Hard Fork
5,368 Listeners
Clearer Thinking with Spencer Greenberg
137 Listeners
Razib Khan's Unsupervised Learning
200 Listeners
No Priors: Artificial Intelligence | Technology | Startups
103 Listeners
Latent Space: The AI Engineer Podcast
64 Listeners
"Econ 102" with Noah Smith and Erik Torenberg
138 Listeners
Complex Systems with Patrick McKenzie (patio11)
101 Listeners
LessWrong posts by zvi
0 Listeners