Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Four visions of Transformative AI success, published by Steve Byrnes on January 17, 2024 on The AI Alignment Forum.

Tl;dr

When people work towards making a good future in regards to Transformative AI (TAI), what's the vision of the future that they have in mind and are working towards?

I'll propose four (caricatured) answers that different people seem to give:

(Vision 1) "Helper AIs",

(Vision 2) "Autonomous AIs",

(Vision 3) "Supercharged biological human brains",

(Vision 4) "Don't build TAI".

For each of these four, I will go through:

the typical assumptions and ideas that these people seem to typically have in mind;

potential causes for concern;

major people, institutions, and research directions associated with this vision.

I'll interject a lot of my own opinions throughout, including a suggestion that, on the current margin, the community should be putting more direct effort into technical work towards contingency-planning for Vision 2.

Oversimplification Warning: This document is full of oversimplifications and caricatures. But hopefully it's a useful starting point for certain purposes.

Jargon Warning: Lots of jargon; my target audience here is pretty familiar with the AGI safety and alignment literature. But DM me if something is confusing and I'll try to fix it.

Vision 1: "Helper AIs" - AIs doing specifically what humans want them to do

1.1 Typical assumptions and ideas

By and large, people in this camp have an assumption that TAI will look, and act, and be trained, much like LLMs, but they'll work better. They also typically have an assumption of slow takeoff, very high compute requirements for powerful AI, and relatively few big actors who are training and running AIs (but many more actors using AI through an API).

There are two common big-picture stories here:

(Less common story) Vision 1 is a vision for the long-term future (

example).

(More common story) Vision 1 is a safe way to ultimately get to Vision 2 (or somewhere else) - i.e., future people with helper AIs can help solve technical problems related to AI alignment, set up better governance and institutions, or otherwise plan next steps.

1.2 Potential causes for concern

There's a risk that somebody makes an autonomous (Vision 2 below) ruthlessly-power-seeking AGI. We need to either prevent that (presumably through governance), or hope that humans-with-AI-helpers can defend themselves against such AGIs.

I'm pretty strongly pessimistic here, and that is probably my biggest single reason for not buying into this vision. But I'm just one guy, not an expert, and I think reasonable people can disagree.

Human bad actors will (presumably) be empowered by AI helpers

Pessimistic take: It's really bad if Vladimir Putin (for example) will have a super-smart loyal AI helper.

Optimistic take: Well, Vladimir Putin's opponents will also have super-smart loyal AI helpers. So maybe that's OK!

"AI slave society" seems kinda bad. Two possible elaborations of that are:

"AI slave society is in fact bad"; or

"Even if AI slave society is not in fact bad, at least some humans will think that it's bad. And then those humans will go try to make Vision 2 autonomous AI happen - whether through advocacy and regulation, or by unilateral action."

There's no sharp line between the helper AIs of Vision 1 and the truly-autonomous AIs of Vision 2. For example, to what extent do the human supervisors really understand what their AI helpers are doing and how? The less the humans understand, the less we can say that the humans are really in control.

One issue here is race-to-the-bottom competitive dynamics: if some humans entrust their AIs with more authority to make fast autonomous decisions for complex inscrutable reasons, then those humans will have a competitive advantage over the humans who don't. Thus they will wind up in control of...