Sign up to save your podcasts
Or
Share Without fundamental advances, misalignment and catastrophe are the default outcomes of training powerful AI
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Without fundamental advances, misalignment and catastrophe are the default outcomes of training powerful AI
A pdf version of this report is available here.
Summary.
In this report we argue that AI systems capable of large scale scientific research will likely pursue unwanted goals and this will lead to catastrophic outcomes. We argue this is the default outcome, even with significant countermeasures, given the current trajectory of AI development.
In Section 1 we discuss the tasks which are the focus of this report. We are specifically focusing on AIs which are capable of dramatically speeding up large-scale novel science; on the scale of the Manhattan Project or curing cancer. This type of task requires a lot of work, and will require the AI to overcome many novel and diverse obstacles.
In Section 2 we argue that an AI which is capable of doing hard, novel science will be approximately consequentialist; that is, its behavior will be well described as taking actions in order [...]
The original text contained 40 footnotes which were omitted from this narration.
---
First published:
January 26th, 2024
Source:
https://www.lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe
---
Narrated by TYPE III AUDIO.
Summary.
In this report we argue that AI systems capable of large scale scientific research will likely pursue unwanted goals and this will lead to catastrophic outcomes. We argue this is the default outcome, even with significant countermeasures, given the current trajectory of AI development.
In Section 1 we discuss the tasks which are the focus of this report. We are specifically focusing on AIs which are capable of dramatically speeding up large-scale novel science; on the scale of the Manhattan Project or curing cancer. This type of task requires a lot of work, and will require the AI to overcome many novel and diverse obstacles.
In Section 2 we argue that an AI which is capable of doing hard, novel science will be approximately consequentialist; that is, its behavior will be well described as taking actions in order [...]
The original text contained 40 footnotes which were omitted from this narration.
---
First published:
January 26th, 2024
Source:
https://www.lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrong
4.8
1111 ratings
A pdf version of this report is available here.
Summary.
In this report we argue that AI systems capable of large scale scientific research will likely pursue unwanted goals and this will lead to catastrophic outcomes. We argue this is the default outcome, even with significant countermeasures, given the current trajectory of AI development.
In Section 1 we discuss the tasks which are the focus of this report. We are specifically focusing on AIs which are capable of dramatically speeding up large-scale novel science; on the scale of the Manhattan Project or curing cancer. This type of task requires a lot of work, and will require the AI to overcome many novel and diverse obstacles.
In Section 2 we argue that an AI which is capable of doing hard, novel science will be approximately consequentialist; that is, its behavior will be well described as taking actions in order [...]
The original text contained 40 footnotes which were omitted from this narration.
---
First published:
January 26th, 2024
Source:
https://www.lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe
---
Narrated by TYPE III AUDIO.
Summary.
In this report we argue that AI systems capable of large scale scientific research will likely pursue unwanted goals and this will lead to catastrophic outcomes. We argue this is the default outcome, even with significant countermeasures, given the current trajectory of AI development.
In Section 1 we discuss the tasks which are the focus of this report. We are specifically focusing on AIs which are capable of dramatically speeding up large-scale novel science; on the scale of the Manhattan Project or curing cancer. This type of task requires a lot of work, and will require the AI to overcome many novel and diverse obstacles.
In Section 2 we argue that an AI which is capable of doing hard, novel science will be approximately consequentialist; that is, its behavior will be well described as taking actions in order [...]
The original text contained 40 footnotes which were omitted from this narration.
---
First published:
January 26th, 2024
Source:
https://www.lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (Curated & Popular)
View all
Conversations with Tyler
2,389 Listeners
Astral Codex Ten Podcast
123 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,136 Listeners
ManifoldOne
87 Listeners
The Jim Rutt Show
251 Listeners
Machine Learning Street Talk (MLST)
87 Listeners
Dwarkesh Podcast
389 Listeners
Hard Fork
5,432 Listeners
Clearer Thinking with Spencer Greenberg
128 Listeners
Razib Khan's Unsupervised Learning
198 Listeners
No Priors: Artificial Intelligence | Technology | Startups
121 Listeners
Latent Space: The AI Engineer Podcast
75 Listeners
"Econ 102" with Noah Smith and Erik Torenberg
145 Listeners
Complex Systems with Patrick McKenzie (patio11)
121 Listeners
LessWrong posts by zvi
1 Listeners