Sign up to save your podcasts
Or
Share “Instrumental Goals Are A Different And Friendlier Kind Of Thing Than Terminal Goals” by johnswentworth, David Lorell
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Instrumental Goals Are A Different And Friendlier Kind Of Thing Than Terminal Goals” by johnswentworth, David Lorell
The Cake
Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that's fine. Baking that cake is my terminal goal.
In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.
My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile [...]
---
Outline:
(00:07) The Cake
(01:50) The Restaurant
(03:50) Happy Instrumental Convergence?
(06:27) All The Way Up
(08:05) Research Threads
---
First published:
January 24th, 2025
Source:
https://www.lesswrong.com/posts/7Z4WC4AFgfmZ3fCDC/instrumental-goals-are-a-different-and-friendlier-kind-of
---
Narrated by TYPE III AUDIO.
Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that's fine. Baking that cake is my terminal goal.
In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.
My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile [...]
---
Outline:
(00:07) The Cake
(01:50) The Restaurant
(03:50) Happy Instrumental Convergence?
(06:27) All The Way Up
(08:05) Research Threads
---
First published:
January 24th, 2025
Source:
https://www.lesswrong.com/posts/7Z4WC4AFgfmZ3fCDC/instrumental-goals-are-a-different-and-friendlier-kind-of
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrong
4.8
1212 ratings
The Cake
Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that's fine. Baking that cake is my terminal goal.
In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.
My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile [...]
---
Outline:
(00:07) The Cake
(01:50) The Restaurant
(03:50) Happy Instrumental Convergence?
(06:27) All The Way Up
(08:05) Research Threads
---
First published:
January 24th, 2025
Source:
https://www.lesswrong.com/posts/7Z4WC4AFgfmZ3fCDC/instrumental-goals-are-a-different-and-friendlier-kind-of
---
Narrated by TYPE III AUDIO.
Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that's fine. Baking that cake is my terminal goal.
In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.
My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile [...]
---
Outline:
(00:07) The Cake
(01:50) The Restaurant
(03:50) Happy Instrumental Convergence?
(06:27) All The Way Up
(08:05) Research Threads
---
First published:
January 24th, 2025
Source:
https://www.lesswrong.com/posts/7Z4WC4AFgfmZ3fCDC/instrumental-goals-are-a-different-and-friendlier-kind-of
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (Curated & Popular)
View all
Making Sense with Sam Harris
26,392 Listeners
Conversations with Tyler
2,423 Listeners
Robert Wright's Nonzero
590 Listeners
Future of Life Institute Podcast
107 Listeners
The Good Fight
904 Listeners
ManifoldOne
92 Listeners
The Prof G Pod with Scott Galloway
5,473 Listeners
Machine Learning Street Talk (MLST)
89 Listeners
Dwarkesh Podcast
488 Listeners
Hard Fork
5,469 Listeners
Clearer Thinking with Spencer Greenberg
132 Listeners
Complex Systems with Patrick McKenzie (patio11)
133 Listeners
The Marginal Revolution Podcast
93 Listeners
Statecraft
34 Listeners
The Last Invention
295 Listeners