Parmy Olson is a technology columnist at Bloomberg and the author of Supremacy, which won the 2024 Financial Times Business Book of the Year. She joins the podcast to discuss the transformation of AI companies from research labs to product businesses. We explore how funding pressures have changed company missions, the role of personalities versus innovation, the challenges faced by safety teams, and power consolidation in the industry.

LINKS:
- Parmy Olson on X (Twitter): https://x.com/parmy
- Parmy Olson’s Bloomberg columns: https://www.bloomberg.com/opinion/authors/AVYbUyZve-8/parmy-olson
- Supremacy (book): https://www.panmacmillan.com/authors/parmy-olson/supremacy/9781035038244

PRODUCED BY:
https://aipodcast.ing

CHAPTERS:
(00:00) Episode Preview
(01:18) Introducing Parmy Olson
(02:37) Personalities Driving AI
(06:45) From Research to Products
(12:45) Has the Mission Changed?
(19:43) The Role of Regulators
(21:44) Skepticism of AI Utopia
(28:00) The Human Cost
(33:48) Embracing Controversy
(40:51) The Role of Journalism
(41:40) Big Tech's Influence

SOCIAL LINKS:

Website: https://podcast.futureoflife.org

Twitter (FLI): https://x.com/FLI_org

Twitter (Gus): https://x.com/gusdocker

LinkedIn: https://www.linkedin.com/company/future-of-life-institute/

YouTube: https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/

Apple: https://geo.itunes.apple.com/us/podcast/id1170991978

Spotify: https://open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP

Adam Gleave is co-founder and CEO of FAR.AI. In this cross-post from The Cognitive Revolution Podcast, he joins to discuss post-AGI scenarios and AI safety challenges. The conversation explores his three-tier framework for AI capabilities, gradual disempowerment concerns, defense-in-depth security, and research on training less deceptive models. Topics include timelines, interpretability limitations, scalable oversight techniques, and FAR.AI’s vertically integrated approach spanning technical research, policy advocacy, and field-building.

LINKS:
Adam Gleave - https://www.gleave.me
FAR.AI - https://www.far.ai
The Cognitive Revolution Podcast - https://www.cognitiverevolution.ai

PRODUCED BY:

https://aipodcast.ing

CHAPTERS:

(00:00) A Positive Post-AGI Vision
(10:07) Surviving Gradual Disempowerment
(16:34) Defining Powerful AIs
(27:02) Solving Continual Learning
(35:49) The Just-in-Time Safety Problem
(42:14) Can Defense-in-Depth Work?
(49:18) Fixing Alignment Problems
(58:03) Safer Training Formulas
(01:02:24) The Role of Interpretability
(01:09:25) FAR.AI's Vertically Integrated Approach
(01:14:14) Hiring at FAR.AI
(01:16:02) The Future of Governance

SOCIAL LINKS:

Website: https://podcast.futureoflife.org

Twitter (FLI): https://x.com/FLI_org

Twitter (Gus): https://x.com/gusdocker

LinkedIn: https://www.linkedin.com/company/future-of-life-institute/

YouTube: https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/

Apple: https://geo.itunes.apple.com/us/podcast/id1170991978

Spotify: https://open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP

Beatrice works at the Foresight Institute running their Existential Hope program. She joins the podcast to discuss the AI pathways project, which explores two alternative scenarios to the default race toward AGI. We examine tool AI, which prioritizes human oversight and democratic control, and d/acc, which emphasizes decentralized, defensive development. The conversation covers trade-offs between safety and speed, how these pathways could be combined, and what different stakeholders can do to steer toward more positive AI futures.

LINKS:
AI Pathways - https://ai-pathways.existentialhope.com
Beatrice Erkers - https://www.existentialhope.com/team/beatrice-erkers

CHAPTERS:
(00:00) Episode Preview
(01:10) Introduction and Background
(05:40) AI Pathways Project
(11:10) Defining Tool AI
(17:40) Tool AI Benefits
(23:10) D/acc Pathway Explained
(29:10) Decentralization Trade-offs
(35:10) Combining Both Pathways
(40:10) Uncertainties and Concerns
(45:10) Future Evolution
(01:01:21) Funding Pilots

PRODUCED BY:
https://aipodcast.ing

SOCIAL LINKS:

Website: https://podcast.futureoflife.org

Twitter (FLI): https://x.com/FLI_org

Twitter (Gus): https://x.com/gusdocker

LinkedIn: https://www.linkedin.com/company/future-of-life-institute/

YouTube: https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/

Apple: https://geo.itunes.apple.com/us/podcast/id1170991978

Spotify: https://open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP

Nate Soares is president of the Machine Intelligence Research Institute. He joins the podcast to discuss his new book "If Anyone Builds It, Everyone Dies," co-authored with Eliezer Yudkowsky. We explore why current AI systems are "grown not crafted," making them unpredictable and difficult to control. The conversation covers threshold effects in intelligence, why computer security analogies suggest AI alignment is currently nearly impossible, and why we don't get retries with superintelligence. Soares argues for an international ban on AI research toward superintelligence.

LINKS:
If Anyone Builds It, Everyone Dies - https://ifanyonebuildsit.com
Machine Intelligence Research Institute -  https://intelligence.org
Nate Soares - https://intelligence.org/team/nate-soares/

PRODUCED BY:

https://aipodcast.ing

CHAPTERS:

(00:00) Episode Preview

(01:05) Introduction and Book Discussion

(03:34) Psychology of AI Alarmism

(07:52) Intelligence Threshold Effects

(11:38) Growing vs Crafting AI

(18:23) Illusion of AI Control

(26:45) Why Iteration Won't Work

(34:35) The No Retries Problem

(38:22) Computer Security Lessons

(49:13) The Cursed Problem

(59:32) Multiple Curses and Complications

(01:09:44) AI's Infrastructure Advantage

(01:16:26) Grading Humanity's Response

(01:22:55) Time Needed for Solutions

(01:32:07) International Ban Necessity

SOCIAL LINKS:

Website: https://podcast.futureoflife.org

Twitter (FLI): https://x.com/FLI_org

Twitter (Gus): https://x.com/gusdocker

LinkedIn: https://www.linkedin.com/company/future-of-life-institute/

YouTube: https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/

Apple: https://geo.itunes.apple.com/us/podcast/id1170991978

Spotify: https://open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP

Luke Drago is the co-founder of Workshop Labs and co-author of the essay series "The Intelligence Curse". The essay series explores what happens if AI becomes the dominant factor of production thereby reducing incentives to invest in people. We explore pyramid replacement in firms, economic warning signs to monitor, automation barriers like tacit knowledge, privacy risks in AI training, and tensions between centralized AI safety and democratization. Luke discusses Workshop Labs' privacy-preserving approach and advises taking career risks during this technological transition.  

"The Intelligence Curse" essay series by Luke Drago & Rudolf Laine: https://intelligence-curse.ai/
Luke's Substack: https://lukedrago.substack.com/
Workshop Labs: https://workshoplabs.ai/

CHAPTERS:
(00:00) Episode Preview
(00:55) Intelligence Curse Introduction
(02:55) AI vs Historical Technology
(07:22) Economic Metrics and Indicators
(11:23) Pyramid Replacement Theory
(17:28) Human Judgment and Taste
(22:25) Data Privacy and Control
(28:55) Dystopian Economic Scenario
(35:04) Resource Curse Lessons
(39:57) Culture vs Economic Forces
(47:15) Open Source AI Debate
(54:37) Corporate Mission Evolution
(59:07) AI Alignment and Loyalty
(01:05:56) Moonshots and Career Advice

Basil Halperin is an assistant professor of economics at the University of Virginia. He joins the podcast to discuss what economic indicators reveal about AI timelines. We explore why interest rates might rise if markets expect transformative AI, the gap between strong AI benchmarks and limited economic effects, and bottlenecks to AI-driven growth. We also cover market efficiency, automated AI research, and how financial markets may signal progress.

• Basil's essay on "Transformative AI, existential risk, and real interest rates": https://basilhalperin.com/papers/agi_emh.pdf
• Read more about Basil's work here: https://basilhalperin.com/

CHAPTERS:

(00:00) Episode Preview

(00:49) Introduction and Background

(05:19) Efficient Market Hypothesis Explained

(10:34) Markets and Low Probability Events

(16:09) Information Diffusion on Wall Street

(24:34) Stock Prices vs Interest Rates

(28:47) New Goods Counter-Argument

(40:41) Why Focus on Interest Rates

(45:00) AI Secrecy and Market Efficiency

(50:52) Short Timeline Disagreements

(55:13) Wealth Concentration Effects

(01:01:55) Alternative Economic Indicators

(01:12:47) Benchmarks vs Economic Impact

(01:25:17) Open Research Questions

SOCIAL LINKS:

Website: https://future-of-life-institute-podcast.aipodcast.ing

Twitter (FLI): https://x.com/FLI_org

Twitter (Gus): https://x.com/gusdocker

LinkedIn: https://www.linkedin.com/company/future-of-life-institute/

YouTube: https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/

Apple Podcasts: https://geo.itunes.apple.com/us/podcast/id1170991978

Spotify: https://open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP

PRODUCED BY: 

https://aipodcast.ing

Esben Kran joins the podcast to discuss why securing AGI requires more than traditional cybersecurity, exploring new attack surfaces, adaptive malware, and the societal shifts needed for resilient defenses. We cover protocols for safe agent communication, oversight without surveillance, and distributed safety models across companies and governments.   

Learn more about Esben's work at: https://blog.kran.ai  

00:00 – Intro and preview 

01:13 – AGI security vs traditional cybersecurity 

02:36 – Rebuilding societal infrastructure for embedded security 

03:33 – Sentware: adaptive, self-improving malware 

04:59 – New attack surfaces 

05:38 – Social media as misaligned AI 

06:46 – Personal vs societal defenses 

09:13 – Why private companies underinvest in security 

13:01 – Security as the foundation for any AI deployment 

14:15 – Oversight without a surveillance state 

17:19 – Protocols for safe agent communication 

20:25 – The expensive internet hypothesis 

23:30 – Distributed safety for companies and governments 

28:20 – Cloudflare’s “agent labyrinth” example 

31:08 – Positive vision for distributed security 

33:49 – Human value when labor is automated 

41:19 – Encoding law for machines: contracts and enforcement 

44:36 – DarkBench: detecting manipulative LLM behavior 

55:22 – The AGI endgame: default path vs designed future 

57:37 – Powerful tool AI 

01:09:55 – Fast takeoff risk 

01:16:09 – Realistic optimism

Benjamin Todd joins the podcast to discuss how reasoning models changed AI, why agents may be next, where progress could stall, and what a self-improvement feedback loop in AI might mean for the economy and society. We explore concrete timelines (through 2030), compute and power bottlenecks, and the odds of an industrial explosion. We end by discussing how people can personally prepare for AGI: networks, skills, saving/investing, resilience, citizenship, and information hygiene.  

Follow Benjamin's work at: https://benjamintodd.substack.com  

Timestamps: 

00:00 What are reasoning models?  

04:04 Reinforcement learning supercharges reasoning 

05:06 Reasoning models vs. agents 

10:04 Economic impact of automated math/code 

12:14 Compute as a bottleneck 

15:20 Shift from giant pre-training to post-training/agents 

17:02 Three feedback loops: algorithms, chips, robots 

20:33 How fast could an algorithmic loop run? 

22:03 Chip design and production acceleration 

23:42 Industrial/robotics loop and growth dynamics 

29:52 Society’s slow reaction; “warning shots” 

33:03 Robotics: software and hardware bottlenecks 

35:05 Scaling robot production 

38:12 Robots at ~$0.20/hour?  

43:13 Regulation and humans-in-the-loop 

49:06 Personal prep: why it still matters 

52:04 Build an information network 

55:01 Save more money 

58:58 Land, real estate, and scarcity in an AI world 

01:02:15 Valuable skills: get close to AI, or far from it 

01:06:49 Fame, relationships, citizenship 

01:10:01 Redistribution, welfare, and politics under AI 

01:12:04 Try to become more resilient  

01:14:36 Information hygiene 

01:22:16 Seven-year horizon and scaling limits by ~2030

On this episode, Calum Chace joins me to discuss the transformative impact of AI on employment, comparing the current wave of cognitive automation to historical technological revolutions. We talk about "universal generous income", fully-automated luxury capitalism, and redefining education with AI tutors. We end by examining verification of artificial agents and the ethics of attributing consciousness to machines.  

Learn more about Calum's work here: https://calumchace.com  

Timestamps:  

00:00:00  Preview and intro 

00:03:02  Past tech revolutions and AI-driven unemployment 

00:05:43  Cognitive automation: from secretaries to every job 

00:08:02  The “peak horse” analogy and avoiding human obsolescence 

00:10:55  Infinite demand and lump of labor 

00:18:30  Fully-automated luxury capitalism 

00:23:31  Abundance economy and a potential employment cliff 

00:29:37  Education reimagined with personalized AI tutors 

00:36:22  Real-world uses of LLMs: memory, drafting, emotional insight 

00:42:56  Meaning beyond jobs: aristocrats, retirees, and kids 

00:49:51  Four futures of superintelligence 

00:57:20  Conscious AI and empathy as a safety strategy 

01:10:55  Verifying AI agents 

01:25:20  Over-attributing vs under-attributing machine consciousness

On this episode, Tom Davidson joins me to discuss the emerging threat of AI-enabled coups, where advanced artificial intelligence could empower covert actors to seize power. We explore scenarios including secret loyalties within companies, rapid military automation, and how AI-driven democratic backsliding could differ significantly from historical precedents. Tom also outlines key mitigation strategies, risk indicators, and opportunities for individuals to help prevent these threats.  

Learn more about Tom's work here: https://www.forethought.org  

Timestamps:  

00:00:00  Preview: why preventing AI-enabled coups matters 

00:01:24  What do we mean by an “AI-enabled coup”? 

00:01:59  Capabilities AIs would need (persuasion, strategy, productivity) 

00:02:36  Cyber-offense and the road to robotized militaries 

00:05:32  Step-by-step example of an AI-enabled military coup 

00:08:35  How AI-enabled coups would differ from historical coups 

00:09:24  Democratic backsliding (Venezuela, Hungary, U.S. parallels) 

00:12:38  Singular loyalties, secret loyalties, exclusive access 

00:14:01  Secret-loyalty scenario: CEO with hidden control 

00:18:10  From sleeper agents to sophisticated covert AIs 

00:22:22  Exclusive-access threat: one project races ahead 

00:29:03  Could one country outgrow the rest of the world? 

00:40:00  Could a single company dominate global GDP? 

00:47:01  Autocracies vs democracies 

00:54:43  Mitigations for singular and secret loyalties 

01:06:25  Guardrails, monitoring, and controlled-use APIs 

01:12:38  Using AI itself to preserve checks-and-balances 

01:24:53  Risk indicators to watch for AI-enabled coups 

01:33:05  Tom’s risk estimates for the next 5 and 30 years 

01:46:50  How you can help – research, policy, and careers