On this special episode of the podcast, Jaan Tallinn talks with Nathan Labenz about Jaan's model of AI risk, the future of AI development, and pausing giant AI experiments.

Timestamps: 
0:00 Nathan introduces Jaan
4:22 AI safety and Future of Life Institute
5:55 Jaan's first meeting with Eliezer Yudkowsky
12:04 Future of AI evolution
14:58 Jaan's investments in AI companies
23:06 The emerging danger paradigm
26:53 Economic transformation with AI
32:31 AI supervising itself 
34:06 Language models and validation
38:49 Lack of insight into evolutionary selection process
41:56 Current estimate for life-ending catastrophe
44:52 Inverse scaling law 
53:03 Our luck given the softness of language models 
55:07 Future of language models
59:43 The Moore's law of mad science
1:01:45 GPT-5 type project
1:07:43 The AI race dynamics
1:09:43 AI alignment with the latest models
1:13:14 AI research investment and safety
1:19:43 What a six-month pause buys us
1:25:44 AI passing the Turing Test
1:28:16 AI safety and risk 
1:32:01 Responsible AI development.
1:40:03 Neuralink implant technology

Special: Jaan Tallinn on Pausing Giant AI Experiments

The Future of Life Institute (FLI) is a nonprofit working to reduce global catastrophic and existential risk from powerful technologies. In particular, FLI focuses on risks from artificial intelligence (AI), biotechnology, nuclear weapons and climate change.

The Institute's work is made up of three main strands: grantmaking for risk reduction, educational outreach, and advocacy within the United Nations, US government and European Union institutions.

FLI has become one of the world's leading voices on the governance of AI having created one of the earliest and most influential sets of governance principles: the Asilomar AI Principles.

Technology

Tech News

Podcasting

Gadgets

Software How-To

Dan Faggella joins the podcast to discuss whether humanity should eventually create AGI, how AI will change power dynamics between institutions, what drives AI progress, and which industries are implementing AI successfully. Find out more about Dan at https://danfaggella.com

Timestamps:
00:00 Value differences in AI 
12:07 Should we eventually create AGI? 
28:22 What is a worthy successor? 
43:19 AI changing power dynamics
59:00 Open source AI 
01:05:07 What drives AI progress? 
01:16:36 What limits AI progress? 
01:26:31 Which industries are using AI?

Dan Faggella on the Race to AGI

Liron Shapira joins the podcast to discuss superintelligence goals, what makes AI different from other technologies, risks from centralizing power, and whether AI can defend us from AI. 

Timestamps:
00:00 Intelligence as optimization-power
05:18 Will LLMs imitate human values? 
07:15 Why would AI develop dangerous goals? 
09:55 Goal-completeness 
12:53 Alignment to which values? 
22:12 Is AI just another technology? 
31:20 What is FOOM? 
38:59 Risks from centralized power 
49:18 Can AI defend us against AI? 
56:28 An Apollo program for AI safety
01:04:49 Do we only have one chance? 
01:07:34 Are we living in a crucial time? 
01:16:52 Would superintelligence be fragile? 
01:21:42 Would human-inspired AI be safe?

Liron Shapira on Superintelligence Goals

Annie Jacobsen joins the podcast to lay out a second by second timeline for how nuclear war could happen. We also discuss time pressure, submarines, interceptor missiles, cyberattacks, and concentration of power. You can find more on Annie's work at https://anniejacobsen.com 

Timestamps:
00:00 A scenario of nuclear war
06:56 Who would launch an attack? 
13:50 Detecting nuclear attacks 
19:37 The first critical seconds 
29:42 Decisions under time pressure
34:27 Lessons from insiders  
44:18 Submarines 
51:06 How did we end up like this? 
59:40 Interceptor missiles 
1:11:25 Nuclear weapons and cyberattacks
1:17:35 Concentration of power

Annie Jacobsen on Nuclear War - a Second by Second Timeline

Katja Grace joins the podcast to discuss the largest survey of AI researchers conducted to date, AI researchers' beliefs about different AI risks, capabilities required for continued AI-related transformation, the idea of discontinuous progress, the impacts of AI from either side of the human-level intelligence threshold, intelligence and power, and her thoughts on how we can mitigate AI risk. Find more on Katja's work at https://aiimpacts.org/.

Timestamps:
0:20 AI Impacts surveys
18:11 What AI will look like in 20 years
22:43 Experts’ extinction risk predictions
29:35 Opinions on slowing down AI development
31:25 AI “arms races”
34:00 AI risk areas with the most agreement
40:41 Do “high hopes and dire concerns” go hand-in-hand?
42:00 Intelligence explosions
45:37 Discontinuous progress
49:43 Impacts of AI crossing the human-level intelligence threshold
59:39 What does AI learn from human culture?
1:02:59 AI scaling
1:05:04 What should we do?

Katja Grace on the Largest Survey of AI Researchers

Holly Elmore joins the podcast to discuss pausing frontier AI, hardware overhang, safety research during a pause, the social dynamics of AI risk, and what prevents AGI corporations from collaborating. You can read more about Holly's work at https://pauseai.info 

Timestamps: 
00:00 Pausing AI
10:23 Risks during an AI pause
19:41 Hardware overhang
29:04 Technological progress 
37:00 Safety research during a pause
54:42 Social dynamics of AI risk
1:10:00 What prevents cooperation? 
1:18:21 What about China? 
1:28:24 Protesting AGI corporations

Special: Jaan Tallinn on Pausing Giant AI Experiments

Download our free app to listen on your phone