Will AI destroy humanity? Most people think that's science fiction. The people actually building it aren't so sure.

Geoffrey Hinton - the Godfather of AI - says there's a 10 to 20 percent chance AI wipes us out. OpenAI’s Sam Altman told Congress his own technology could 'cause significant harm to the world.'

Our guest, Malo Bourgon, CEO of the Machine Intelligence Research Institute (MIRI) has been warning about this for two decades. He says a machine doesn't need to be sentient to become a global risk. And today's safety measures? Nowhere near enough.

Can we build an off switch for a machine that's smarter than us?

Malo tells hosts Autria Godfrey and Laila Rivzi that he thinks we’re on the path to building systems that are radically smarter than us, speeding faster in a race to build systems that we don’t understand and that we don’t know how to control that “could end up with none of us around to see the future we could have built instead.”

While he’s not worried that current AI models are existentially dangerous, Malo agrees that some emergent behaviors, like the deception we’re seeing in smarter general systems, suggest that if we continue to scale towards superintelligence, they’ll only have a bigger impact.

The trio discuss whether it would even be possible to build in a safeguard, like an off switch. Malo considers this a losing battle if we wait too long and even then, it would be better if we built a broader system that allows us to not get into that position in the first place.

Some harms and disruptions from AI are already happening. Malo suggests that we need to find some way to have coordinated action globally, even among adversaries. When it comes to existential risk, he draws a comparison to the nuclear arms race and the cold war, and how, thanks to treaties and agreements and some luck along the way, we’re still here.

Autria asks about specific ways AI can lead to a catastrophic, apocalyptic ending, including disruptions to the food chain, the creation of bioweapons, mass unemployment and faltering economies. Malo says even if we solve those, there remains the core danger which comes from a misalignment of the values and goals of the superintelligence we build and our own. It doesn’t have to be evil, it just has to” care about weird, other things that aren’t the things we care about and it wants to pursue those things” with an indifference to our existence.

So what would convince Malo that AI is safe? Changing how we create AI systems, he says, from growing them “in a very brute force way” the way we do now to crafting them with a more principled sense of what we’re trying to do to make them safe.

Finally, Malo suggests that the claim that “the people running these companies are, you know, bad, immoral people" isn’t quite right. He says that most of “these people aren't actually villains, they're normal people who are kind of trying to do the right thing but they're in a really bad situation. But I also think they're also doing a bunch of bad stuff.”

What about you? We want to know what you think. How concerned should we be about a doomsday scenario? How concerned are you? Tell us in the comments below,

CHAPTERS:

00:00 - Profits, Power, and AI Risk

02:36 - Are We Racing Toward Extinction?

03:33 - Are Big Tech’s Predictions of AGI Accurate?

04:50 - Are AI Models Exhibiting Signs of Understanding?

05:01 - Deception in Today’s Models

07:51 - Can’t We Just Build an AI Off-Switch?

10:10 - What Can We Actually Do?

11:06 - Harms Are Already Here – More Over The Horizon

12:00 - Global Race to AI Superintelligence

13:00 - Parallels to Nuclear Arms Race

15:14 - Pathways to Catastrophic Risk

16:14 - The Boss Fight – Misalignment As The Core Danger

18:51 - What Would “Safe” Look Like?

19:40 - Rapid Fire Questions for Malo Bourgon

19:44 - What Does Your Field Get Wrong?

20:53 - Where Do People Draw The Line On AI In Their Lives?

22:24 - Post-Interview Reflections

24:44 - What Do You Think: Doomsday Scenario or Not?

ChatGPT has 800 million users. OpenAI is valued at $500 billion. But our guest today says the whole thing is a scam. Professor Emily Bender, author of “The AI Con” and Director of the Computational Linguistics Laboratory at University of Washington, argues Artificial Intelligence is just a broad marketing term and “AI" is just a label for unrelated tech - creating a false sense of an inevitable, God-like entity.

Is she a prophet... or is she just wrong?

We’ll ask our questions for Professor Bender in the episode, but if you’ve got questions for us, throw them into the comments below!

Hosts Autria Godfrey and Laila Rizvi start by asking Emily whether AI is intelligent enough to replace humans. Emily says studies indicating that AI models are cheating, blackmailing, and playing dumb when they know they’re being tested don’t stand up. She says it’s elaborate interactive fiction, and that Anthropic’s “research” isn’t peer reviewed – basically, no more than blog posts. LLM training includes language that looks like introspection, so systems can output language that looks like introspection even though they have no capacity to actually engage in introspection.

Emily suggests that replacing interns and entry level workers with AI short-circuits the process of training future leaders. She describes how AI systems exploit the Global South, with difficult psychological conditions and compensation so low it creates, as Autria suggests, the next generation of sweat shops.

When it comes to AI 2027 and whether AI poses an existential threat, Emily says it’s just a case of “Big Tech Fan Fiction” from the same shared world as the thinking of Nick Bostrom and the Effective Altruist movement.

What about claims by Anthropic that Claude Code wrote the code for Claude Cowork? Emily doubts those claims, explaining that those systems have no agency and require input to do something.

Although Emily doesn’t buy into claims of near-term existential risk, AI is creating labor and environmental harm on local levels if not global ones, often with a lack of transparency.

What about arguments like those by Nobel Prize winner Geoffrey Hinton that suggest LLMs understand meaning and can mirror how humans operate? Emily says that given his background and specific knowledge of how these systems are built, he “really ought to know better.” She explains that unless we have access to the training data actually used on these systems, we can’t know that they are actually understanding concepts without explicit training.

After Professor Bender leaves, Autria and Laila discuss whether Professor Bender’s dismissal of some of the data Laila presented is appropriate or incorrect.

CHAPTERS:

00:00 - Is AI Hype a Scam?

01:33 - AI: Existential Risk or Theater?

02:02 - Dario Amodei and Demis Hassabis At Davos: 1-2 years Until AI Is a Risk

02:50 - Revolution or Con?

03:07 - How intelligent is AI, really? We ask Emily Bender

03:30 - Is AI Intelligent Enough to Replace Humans? Emily Bender Says No!

04:24 - “Cheating” Models and False Agency

06:32 - Will AI Take Our Jobs or Just Make Them Crappier?

06:43 - AI and the Career Ladder Problem

07:54 - Are AI Systems Exploiting Data Workers in the Global South?

08:18 - The Hidden Human Labor of AI

10:47 - AI 2027 and Big Tech Fan Fiction?

12:29 - Are LLMs like Claude Really Writing Their Own Code?

13:45 - Does AI Code Itself?

14:41 - Does AI Need to Be All-Powerful to Pose an Existential Risk?

15:44 - Environmental and Labor Harms

16:35 - Is AI Power and Water Consumption As Bad As Some People Claim?

17:41 - If AI’s Importance to Humanity Is Overhyped, Why Do So Many Believe It?

17:52 - Why the Hype Worked

18:48 - Can Neural Networks Mirror Human Neurology?

21:02 - Geoffrey Hinton and “Understanding”

22:07 - What Is AI Actually Good For?

23:23 - Questions for Professor Bender

23:36 - Is AGI Inevitable?

24:08 - Where Do Humans Draw the Line?

25:28 - After the Interview: Who’s Right?

27:34 - What Do You Think: Doomsday or Hype?

Diplomacy used to be about treaties and territory – now it seems it's more about data, algorithms, and the companies that control them. At Donald Trump’s inauguration, Silicon Valley’s most powerful figures stood steps away, a sign that Big Tech now sits at the centre of global power.

Tech companies pervade everyday life and wield power once reserved for nation states. Are the people in charge of global power those elected to office or those appointed to positions within those companies?

To explore how AI is reshaping diplomacy, from negotiation and representation to influence operations and disinformation, hosts Autria Godfrey, Stephen Horn, and Laila Rizvi interview Dr. Jennifer Cassidy, AI & Diplomacy, University of Oxford, about:

• How AI is transforming diplomacy’s core functions
• Why Big Tech now rivals governments in geopolitical influence
• The rise of “digital sovereigns” and private power
• When former political leaders move into tech, where accountability goes
• Democratic versus authoritarian uses of AI
• Why global AI governance is still largely non-binding

For Dr. Cassidy, diplomacy rests on three, timeless pillars: communication, representation, and negotiation. AI “is not demolishing these pillars, but quietly rewiring the architecture that holds them together… Predictive analysis now allows ministries to read the global mood” almost in real-time. The United Nations and the World Bank use AI models that monitor food prices, rainfall patterns, and social media data to anticipate instability “up to 6 weeks before that instability might actually break out.” NATO employs machine learning to map Russian disinformation. “What we’re seeing here is the move from reactive diplomacy… to anticipatory diplomacy.”

One of the most pressing questions is whose AI is being used to create “sovereign diplomatic AI systems.” France and the EU train their AI on Mistral, a French company. US AI models are OpenAI's and Anthropic's. Microsoft's Azure Cloud hosts data for NATO and national governments.

These companies have become “digital sovereigns” – private actors who control the three levers of power that were once defined by the state: information, infrastructure and interpretation.

Former politicians like Nick Clegg (Meta) and Rishi Sunak (Microsoft) represent a “circuit of influence” where “experience, access, and authority are just flowing continuously between capitals and campuses in Silicon Valley.” While “democracies do need experienced voices helping to steer the tech transition,” we must ensure that “when the expertise moves, accountability moves with it.”

What about bad actors using AI? Jennifer says we’ve seen this in elections in the US and the world. In China, “predictive policing algorithms are tracking not just where crime might occur, but who might commit it… Authoritarian regimes are combining facial recognition, travel data, and digital behaviour into vast surveillance scores.” It is “digital authoritarianism in its most refined form… controlled by prediction, rather than force.”

Dr. Cassidy concludes, “We have a very, very, very long way to go regarding the governance and structure of, and frameworks for AI… a difficult task… that has to be done.”

What’s your take? Share your thoughts in the comments and subscribe for more on AI, geopolitics and global power.

CHAPTERS

00:00 Tech, Trump and the New Global Power Game

01:26 Do Tech Giants Now Run Foreign Policy?

04:00 How AI Is Reshaping Diplomacy?

06:37 Why Nations Are Building Their Own AI Models

09:18 Have Big Tech Companies Become Sovereigns?

12:33 From Prime Minister to Big Tech: The Revolving Door

16:46 AI Power Politics Beyond the West

19:43 AI for Good or Digital Authoritarianism?

22:09 Who Sets the Rules for AI?

24:48 Closing Thoughts with Dr. Jennifer Cassidy

25:05 Debrief: Authoritarian Drift and Regulation Fights

27:13 AI Ministers, Echo Chambers and What Comes Next

Quantum computing may finally be ready for the real world - and it could power the next wave of AI. In this episode of Agents of Tech we sit down with Oxford Quantum Circuits (OQC) CEO Gerald Mullally to explore how OQC is integrating quantum computers into data-centres in London, Tokyo and New York.

Larry Ellison built Oracle into a cornerstone of the modern tech economy. Now he is making a $2.5 billion bet on Oxford, backing the Ellison Institute of Technology at Oxford to fuse AI, medicine and sustainability in one global hub.

In this episode of Agents of Tech, Autria Godfrey, Stephen Horn and Laila Rizvi sit down in Oxford with Professor Santa Ono, Global President of the Ellison Institute of Technology (EIT), to ask a simple question: Can Oxford really become Europe’s Silicon Valley?

We explore:

- Why Ellison chose Oxford and the UK over Chicago or California

- How EIT plans to recruit 7,000 world class scientists and double Oxford’s research base

- The model of science-led capitalism and why commercialization is central to Ellison’s vision

- The UK’s unique advantage in health data and biobanks (NHS data, UK Biobank, Protein Data Bank)

- How AI, machine learning and robotics will change drug discovery, pandemics and healthcare

- The relationship between EIT and Oracle, and how independent the institute really is

- Parallels and contrasts with the Bill & Melinda Gates Foundation model of philanthropy

- What this means for the UK’s role between the US and China in the global innovation race

Professor Ono explains why he believes the UK is now one of the best places in the world to build AI-driven science: from single-payer health data to a fast-growing ecosystem of serial entrepreneurs. He also addresses questions about data privacy, ethics, bioterrorism risks and public concerns about American tech money in historic British institutions.

If you care about:

- How AI and health data will reshape medicine

- Whether Oxford and Cambridge can anchor Europe’s answer to Silicon Valley

- What it really takes to build a global science and technology campus at scale

…this conversation is for you.

Tell us in the comments: Do you think Ellison’s Oxford gamble is a bold new model for global science, or another moonshot that will be hard to scale?

CHAPTERS

00:00 Larry Ellison’s $2.5B bet on Oxford
00:35 Agents of Tech intro
01:22 Why Oxford?
02:45 Interview begins: Santa Ono
03:01 Ellison’s vision for EIT
05:11 Scaling talent and entrepreneurship
05:53 Science capitalism vs traditional philanthropy
07:52 Why base EIT in the UK
10:38 NHS data, privacy and AI concerns
12:55 AI’s impact on jobs and drug discovery
15:12 Commercialisation and scientific breakthroughs
17:38 Building a new global research hub
20:26 AI geopolitics and the UK’s role
21:03 EIT as a global model
22:50 Interview ends
23:01 Post-interview reflections
24:41 Closing and invitation to Larry Ellison

Is the U.S. LOSING the AI race to China?

China and the U.S. are neck and neck in the AI race for global dominance. Former OpenAI board member Helen Toner (now at Georgetown’s CSET) joins us in Washington, D.C. to break down China vs U.S. strategies—open-source diffusion vs big tech global dominance— and what “winning” actually means.

Helen has recently spent time in China and works at the center of U.S. AI policy—offering a rare inside view of both ecosystems and who’s truly ahead.

Helen explains:

- Who’s ahead right now and how to measure it (frontier AI vs adoption/diffusion)

- Open-source vs closed: DeepSeek, Qwen, Kimi, Gemma, Llama vs OpenAI, Anthropic, Google

- Compute & chips: NVIDIA dependence, export controls, and why compute concentration matters

- AGI timelines: whether “AI 2027” holds up and why short timelines cooled after GPT-5

- “AI+” strategy: applying AI to manufacturing, healthcare, and finance vs pure frontier bragging rights

- What governments should do now: transparency, auditing, AI literacy, and measurement science

Who do you think is winning and WHY – China or the U.S.? Drop one evidence-backed reason (links welcome). We’ll pin the best reply.

Don’t forget to like and subscribe for more unfiltered conversations on AI, tech, and society.

Chapters

00:00 – Two strategies, one AI race

01:00 – Open-source China vs Big-Tech USA

03:37 – Not one race: choose your finish line

04:04 – Who’s actually open? DeepSeek, Qwen/Kimi, Llama, Gemma, GPT-OSS

06:26 – Frontier bragging rights vs real-world adoption

07:46 – China’s “AI Plus” play (AI + industry)

10:06 – Is the US still ahead at the frontier?

12:04 – GPT-5 reality check & AGI timelines

20:58 – Compute decides: chips, export controls, auto-ML engineers

23:04 – What we need now: transparency, audits, AI literacy

28:02 – Standards in practice: de-facto beats de-jure

30:56 – Next 5 years: closed peaks, open bow wave

37:55 – Final take: which path wins?

#OpenAI #HelenToner #ai #GPT5 #OpenSource #podcast #China #DeepSeek

Is the U.S. LOSING the AI race to China?

China and the U.S. are neck and neck in the AI race for global dominance. Former OpenAI board member Helen Toner (now at Georgetown’s CSET) joins us in Washington, D.C. to break down China vs U.S. strategies—open-source diffusion vs big tech global dominance— and what “winning” actually means.

Helen has recently spent time in China and works at the center of U.S. AI policy—offering a rare inside view of both ecosystems and who’s truly ahead.

Helen explains:

- Who’s ahead right now and how to measure it (frontier AI vs adoption/diffusion)

- Open-source vs closed: DeepSeek, Qwen, Kimi, Gemma, Llama vs OpenAI, Anthropic, Google

- Compute & chips: NVIDIA dependence, export controls, and why compute concentration matters

- AGI timelines: whether “AI 2027” holds up and why short timelines cooled after GPT-5

- “AI+” strategy: applying AI to manufacturing, healthcare, and finance vs pure frontier bragging rights

- What governments should do now: transparency, auditing, AI literacy, and measurement science

Who do you think is winning and WHY – China or the U.S.? Drop one evidence-backed reason (links welcome). We’ll pin the best reply.

Don’t forget to like and subscribe for more unfiltered conversations on AI, tech, and society.

Chapters

00:00 – Two strategies, one AI race

01:00 – Open-source China vs Big-Tech USA

03:37 – Not one race: choose your finish line

04:04 – Who’s actually open? DeepSeek, Qwen/Kimi, Llama, Gemma, GPT-OSS

06:26 – Frontier bragging rights vs real-world adoption

07:46 – China’s “AI Plus” play (AI + industry)

10:06 – Is the US still ahead at the frontier?

12:04 – GPT-5 reality check & AGI timelines

20:58 – Compute decides: chips, export controls, auto-ML engineers

23:04 – What we need now: transparency, audits, AI literacy

28:02 – Standards in practice: de-facto beats de-jure

30:56 – Next 5 years: closed peaks, open bow wave

37:55 – Final take: which path wins?

#OpenAI #HelenToner #ai #GPT5 #OpenSource #podcast #China #DeepSeek

Has OpenAI made the WRONG bet?

Gary Marcus argues that OpenAI is taking the wrong approach - and the AI bubble is real. We walk through why GPT-5 underwhelmed, where the scaling paradigm breaks, and what might really get us to AGI.

Gary explains:
– Why the economics don’t add up for current AI
– Why GPT-5 isn’t as good as expected
– The core LLM limitations and why the scaling paradigm fails
– Why AI won’t take your job in the near term
– A practical path to AGI (hybrid / neuro-symbolic, world models)

We also debate whether investors are over- or under-valuing AI, what productivity gains are real, and how long it will take before AI truly replaces jobs.

References discussed in this episode:

– Gary Marcus, The Algebraic Mind: Integrating Connectionism and Cognitive Science (2001)
– Gary Marcus, “Deep Learning Is Hitting a Wall” (Nautilus, 2022)
– Gary Marcus, The Next Decade in AI (arXiv, 2020)
– Mike Dash, Tulipomania

Do you think Gary is right—or are we just getting started? Drop one strong piece of evidence either way (links welcome). We’ll pin the best reply. Don’t forget to like and subscribe for more unfiltered conversations on AI, tech, and society.

Chapters
00:00 – Is AI in a bubble?
00:30 – AI hype vs. reality
01:10 – GPT-5 launch: disappointment or progress?
03:50 – Can AI be both a revolution and a bubble?
05:40 – Productivity gains and investment hype
08:00 – Gary Marcus joins the conversation
09:30 – Why Gary calls himself a skeptic
11:15 – GPT-5 and the limits of scaling
14:00 – Financial reality of large language models
17:20 – “Deep Learning Is Hitting a Wall”
19:00 – Why hallucinations won’t go away
21:00 – Neuro-symbolic AI explained
24:00 – Building world models for AI
27:00 – Are AI valuations sustainable?
29:30 – Lessons from Tulipomania
31:30 – Will AI take all our jobs?
36:00 – What comes next for AI research
38:30 – Final thoughts

#OpenAI #GaryMarcus #ai #GPT5 #scaling #podcast

AI agents can’t easily talk to each other. To solve this, Cisco has joined forces with Google, Dell, Oracle, and Red Hat to launch AGNTCY, the open-source Internet of Agents. It’s designed so agents from different organizations can discover, trust, and work together - and it’s now donated to the Linux Foundation for neutral governance.

Vijoy Pandey, Senior VP and GM of Outshift by Cisco, explains how AGNTCY tackles interoperability, why secure messaging (via SLIM) matters, and how enterprises are already using multi-agent systems in the real world. We also look ahead to quantum networking - the next frontier Cisco and its partners are preparing for, connecting processors and data centers across vast distances.

👉 Tell us in the comments how you see open-source agents changing your industry.

Visit AGNTCY.org
Visit Outshift.com

Chapters
00:00 Cold open
00:34 Welcome and hosts
01:45 Why interoperability is the bottleneck
03:10 What AGNTCY enables
05:55 Trust, identity, and evaluation
12:40 SLIM and secure agent messaging
16:45 Real-world deployments
20:10 From pilots to production
21:55 Open source and Linux Foundation
25:20 Quantum networking outlook
28:50 Final thoughts and CTA

Guest:
Vijoy Pandey, SVP and GM, Outshift by Cisco

#AI #Agents #Cisco #AGNTCY #Interoperability #OpenSource #LinuxFoundation #Quantum #AgenticAI #Outshift

AI is changing the way we do biology. But can it crack one of the toughest challenges in science — drug discovery?

In this episode of Agents of Tech, we speak with Professor Charlotte Deane, MBE, University of Oxford, one of the UK’s leading computational biologists. As a co-lead of OpenBind, Charlotte is helping build the global data infrastructure needed to accelerate drug discovery with AI.

From the role of open science to the cultural gap in innovation between the US and UK, Charlotte explains why the UK, despite its size, continues to punch well above its weight in AI and computational biology.

📍 Recorded live at ISMB/ECCB in Liverpool, the world’s leading computational biology conference.

🔔 Subscribe to Agents of Tech for more conversations with the people shaping the future of AI, science, and society: https://www.youtube.com/@AgentsOfTech

#AI #DrugDiscovery #CharlotteDeane #UKAI #Biology #AgentsOfTech

00:00 Intro: AI’s toughest challenge – drug discovery

00:27 Welcome to Agents of Tech at ISMB/ECCB, Liverpool

01:00 Why is drug discovery so hard for AI?

02:00 The UK and AI: overlooked but world-class

05:57 Innovation culture: risk-taking in the UK vs US

08:00 Introducing Professor Charlotte Deane

09:12 What is the OpenBind consortium?

13:00 Pharma data vs open science data

15:59 How the UK punches above its weight in AI

19:20 Training the next generation of AI-literate scientists

22:00 How AI will change the way science is done

25:00 Can AI ask better scientific questions?

27:40 How AI is changing drug discovery workflows

32:00 Final thoughts from Charlotte Deane

34:00 Host debrief: UK AI, OpenBind, and the future of biology