Will AI destroy humanity? Most people think that's science fiction. The people actually building it aren't so sure.

Geoffrey Hinton - the Godfather of AI - says there's a 10 to 20 percent chance AI wipes us out. OpenAI’s Sam Altman told Congress his own technology could 'cause significant harm to the world.'

Our guest, Malo Bourgon, CEO of the Machine Intelligence Research Institute (MIRI) has been warning about this for two decades. He says a machine doesn't need to be sentient to become a global risk. And today's safety measures? Nowhere near enough.

Can we build an off switch for a machine that's smarter than us?

Malo tells hosts Autria Godfrey and Laila Rivzi that he thinks we’re on the path to building systems that are radically smarter than us, speeding faster in a race to build systems that we don’t understand and that we don’t know how to control that “could end up with none of us around to see the future we could have built instead.”

While he’s not worried that current AI models are existentially dangerous, Malo agrees that some emergent behaviors, like the deception we’re seeing in smarter general systems, suggest that if we continue to scale towards superintelligence, they’ll only have a bigger impact.

The trio discuss whether it would even be possible to build in a safeguard, like an off switch. Malo considers this a losing battle if we wait too long and even then, it would be better if we built a broader system that allows us to not get into that position in the first place.

Some harms and disruptions from AI are already happening. Malo suggests that we need to find some way to have coordinated action globally, even among adversaries. When it comes to existential risk, he draws a comparison to the nuclear arms race and the cold war, and how, thanks to treaties and agreements and some luck along the way, we’re still here.

Autria asks about specific ways AI can lead to a catastrophic, apocalyptic ending, including disruptions to the food chain, the creation of bioweapons, mass unemployment and faltering economies. Malo says even if we solve those, there remains the core danger which comes from a misalignment of the values and goals of the superintelligence we build and our own. It doesn’t have to be evil, it just has to” care about weird, other things that aren’t the things we care about and it wants to pursue those things” with an indifference to our existence.

So what would convince Malo that AI is safe? Changing how we create AI systems, he says, from growing them “in a very brute force way” the way we do now to crafting them with a more principled sense of what we’re trying to do to make them safe.

Finally, Malo suggests that the claim that “the people running these companies are, you know, bad, immoral people" isn’t quite right. He says that most of “these people aren't actually villains, they're normal people who are kind of trying to do the right thing but they're in a really bad situation. But I also think they're also doing a bunch of bad stuff.”

What about you? We want to know what you think. How concerned should we be about a doomsday scenario? How concerned are you? Tell us in the comments below,

CHAPTERS:

00:00 - Profits, Power, and AI Risk

02:36 - Are We Racing Toward Extinction?

03:33 - Are Big Tech’s Predictions of AGI Accurate?

04:50 - Are AI Models Exhibiting Signs of Understanding?

05:01 - Deception in Today’s Models

07:51 - Can’t We Just Build an AI Off-Switch?

10:10 - What Can We Actually Do?

11:06 - Harms Are Already Here – More Over The Horizon

12:00 - Global Race to AI Superintelligence

13:00 - Parallels to Nuclear Arms Race

15:14 - Pathways to Catastrophic Risk

16:14 - The Boss Fight – Misalignment As The Core Danger

18:51 - What Would “Safe” Look Like?

19:40 - Rapid Fire Questions for Malo Bourgon

19:44 - What Does Your Field Get Wrong?

20:53 - Where Do People Draw The Line On AI In Their Lives?

22:24 - Post-Interview Reflections

24:44 - What Do You Think: Doomsday Scenario or Not?