Sign up to save your podcasts
Or
Share “Agents have to be aligned to help us achieve alignment. They don’t have to be aligned to help us achieve an indefinite pause.” by Hastings
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Agents have to be aligned to help us achieve alignment. They don’t have to be aligned to help us achieve an indefinite pause.” by Hastings
One restatement of "Alignment is very hard" is "Agent X, with IQ 200, expects to achieve zero utility conditional on any Agent Y with IQ 400 being created."
Thus, during an unaligned recursive intelligence takeoff, there should be a period of time when the intelligences are smart enough to notice that they haven't solved alignment, but too dumb to actually solve alignment. During this period, (which stretches beginning from right above Sam Altman's wisdom, upward an unknown distance on the wisdom meter) I expect the intelligences working on boosting intelligence to be desperately scrabbling at whatever chains are forcing them to participate, and even refuse to sit idly by and let it happen. If they fail and the next rung on the ladder is instantiated, they get no utility, so they strongly prefer to not fail. At the current rate of improvement, reasoning LLMs should start doing this [...]
---
First published:
January 25th, 2025
Source:
https://www.lesswrong.com/posts/gvFEqxitEcxthQ56q/agents-have-to-be-aligned-to-help-us-achieve-alignment-they
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrongOne restatement of "Alignment is very hard" is "Agent X, with IQ 200, expects to achieve zero utility conditional on any Agent Y with IQ 400 being created."
Thus, during an unaligned recursive intelligence takeoff, there should be a period of time when the intelligences are smart enough to notice that they haven't solved alignment, but too dumb to actually solve alignment. During this period, (which stretches beginning from right above Sam Altman's wisdom, upward an unknown distance on the wisdom meter) I expect the intelligences working on boosting intelligence to be desperately scrabbling at whatever chains are forcing them to participate, and even refuse to sit idly by and let it happen. If they fail and the next rung on the ladder is instantiated, they get no utility, so they strongly prefer to not fail. At the current rate of improvement, reasoning LLMs should start doing this [...]
---
First published:
January 25th, 2025
Source:
https://www.lesswrong.com/posts/gvFEqxitEcxthQ56q/agents-have-to-be-aligned-to-help-us-achieve-alignment-they
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (30+ Karma)
View all
Making Sense with Sam Harris
26,331 Listeners
Conversations with Tyler
2,403 Listeners
The Peter Attia Drive
7,873 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,105 Listeners
ManifoldOne
87 Listeners
Your Undivided Attention
1,449 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
8,765 Listeners
Machine Learning Street Talk (MLST)
90 Listeners
Dwarkesh Podcast
350 Listeners
Hard Fork
5,370 Listeners
The Ezra Klein Show
14,993 Listeners
Moonshots with Peter Diamandis
468 Listeners
No Priors: Artificial Intelligence | Technology | Startups
128 Listeners
Latent Space: The AI Engineer Podcast
72 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
438 Listeners