Sign up to save your podcasts
Or
Share “Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?” by Jesse Richardson, Yoshua Bengio, dwk, mattmacdermott
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?” by Jesse Richardson, Yoshua Bengio, dwk, mattmacdermott
A new paper by Yoshua Bengio and the Safe Artificial Intelligence For Humanity (SAIFH) team argues that the current push towards building generalist AI agents presents catastrophic risks, creating a need for more caution and an alternative approach. We propose such an approach in the form of Scientist AI, a non-agentic AI system that aims to be the foundation for safe superintelligence. (Note that this paper is intended for a broad audience, including readers unfamiliar with AI safety.)
Abstract
The leading AI companies are increasingly focused on building generalist AI agents—systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. We discuss how these risks arise from current AI [...]
---
Outline:
(00:42) Abstract
(02:42) Executive Summary
(02:47) Highly effective AI without agency
(09:51) Mapping out ways of losing control
(15:24) The Scientist AI research plan
(20:21) Career Opportunities at SAIFH
---
First published:
February 24th, 2025
Source:
https://www.lesswrong.com/posts/p5gBcoQeBsvsMShvT/superintelligent-agents-pose-catastrophic-risks-can
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrongA new paper by Yoshua Bengio and the Safe Artificial Intelligence For Humanity (SAIFH) team argues that the current push towards building generalist AI agents presents catastrophic risks, creating a need for more caution and an alternative approach. We propose such an approach in the form of Scientist AI, a non-agentic AI system that aims to be the foundation for safe superintelligence. (Note that this paper is intended for a broad audience, including readers unfamiliar with AI safety.)
Abstract
The leading AI companies are increasingly focused on building generalist AI agents—systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. We discuss how these risks arise from current AI [...]
---
Outline:
(00:42) Abstract
(02:42) Executive Summary
(02:47) Highly effective AI without agency
(09:51) Mapping out ways of losing control
(15:24) The Scientist AI research plan
(20:21) Career Opportunities at SAIFH
---
First published:
February 24th, 2025
Source:
https://www.lesswrong.com/posts/p5gBcoQeBsvsMShvT/superintelligent-agents-pose-catastrophic-risks-can
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (30+ Karma)
View all
Making Sense with Sam Harris
26,334 Listeners
Conversations with Tyler
2,399 Listeners
The Peter Attia Drive
7,817 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,107 Listeners
ManifoldOne
87 Listeners
Your Undivided Attention
1,453 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
8,761 Listeners
Machine Learning Street Talk (MLST)
90 Listeners
Dwarkesh Podcast
353 Listeners
Hard Fork
5,356 Listeners
The Ezra Klein Show
15,023 Listeners
Moonshots with Peter Diamandis
464 Listeners
No Priors: Artificial Intelligence | Technology | Startups
128 Listeners
Latent Space: The AI Engineer Podcast
73 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
433 Listeners