Sign up to save your podcasts
Or
Share “Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?” by Jesse Richardson, Yoshua Bengio, dwk, mattmacdermott
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?” by Jesse Richardson, Yoshua Bengio, dwk, mattmacdermott
A new paper by Yoshua Bengio and the Safe Artificial Intelligence For Humanity (SAIFH) team argues that the current push towards building generalist AI agents presents catastrophic risks, creating a need for more caution and an alternative approach. We propose such an approach in the form of Scientist AI, a non-agentic AI system that aims to be the foundation for safe superintelligence. (Note that this paper is intended for a broad audience, including readers unfamiliar with AI safety.)
Abstract
The leading AI companies are increasingly focused on building generalist AI agents—systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. We discuss how these risks arise from current AI [...]
---
Outline:
(00:42) Abstract
(02:42) Executive Summary
(02:47) Highly effective AI without agency
(09:51) Mapping out ways of losing control
(15:24) The Scientist AI research plan
(20:21) Career Opportunities at SAIFH
---
First published:
February 24th, 2025
Source:
https://www.lesswrong.com/posts/p5gBcoQeBsvsMShvT/superintelligent-agents-pose-catastrophic-risks-can
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrongA new paper by Yoshua Bengio and the Safe Artificial Intelligence For Humanity (SAIFH) team argues that the current push towards building generalist AI agents presents catastrophic risks, creating a need for more caution and an alternative approach. We propose such an approach in the form of Scientist AI, a non-agentic AI system that aims to be the foundation for safe superintelligence. (Note that this paper is intended for a broad audience, including readers unfamiliar with AI safety.)
Abstract
The leading AI companies are increasingly focused on building generalist AI agents—systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. We discuss how these risks arise from current AI [...]
---
Outline:
(00:42) Abstract
(02:42) Executive Summary
(02:47) Highly effective AI without agency
(09:51) Mapping out ways of losing control
(15:24) The Scientist AI research plan
(20:21) Career Opportunities at SAIFH
---
First published:
February 24th, 2025
Source:
https://www.lesswrong.com/posts/p5gBcoQeBsvsMShvT/superintelligent-agents-pose-catastrophic-risks-can
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (30+ Karma)
View all
Making Sense with Sam Harris
26,345 Listeners
Conversations with Tyler
2,397 Listeners
The Peter Attia Drive
8,002 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,119 Listeners
ManifoldOne
90 Listeners
Your Undivided Attention
1,501 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
9,263 Listeners
Machine Learning Street Talk (MLST)
90 Listeners
Dwarkesh Podcast
426 Listeners
Hard Fork
5,463 Listeners
The Ezra Klein Show
15,405 Listeners
Moonshots with Peter Diamandis
517 Listeners
No Priors: Artificial Intelligence | Technology | Startups
125 Listeners
Latent Space: The AI Engineer Podcast
71 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
469 Listeners