Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When discussing AI risks, talk about capabilities, not intelligence, published by Victoria Krakovna on August 11, 2023 on The AI Alignment Forum.

Public discussions about catastrophic risks from general AI systems are often derailed by using the word "intelligence". People often have different definitions of intelligence, or associate it with concepts like consciousness that are not relevant to AI risks, or dismiss the risks because intelligence is not well-defined. I would advocate for using the term "capabilities" or "competence" instead of "intelligence" when discussing catastrophic risks from AI, because this is what the concerns are really about. For example, instead of "superintelligence" we can refer to "super-competence" or "superhuman capabilities".

When we talk about general AI systems posing catastrophic risks, the concern is about losing control of highly capable AI systems. Definitions of general AI that are commonly used by people working to address these risks are about general capabilities of the AI systems:

PASTA definition: "AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement".

Legg-Hutter definition: "An agent's ability to achieve goals in a wide range of environments".

We expect that AI systems that satisfy these definitions would have general capabilities including long-term planning, modeling the world, scientific research, manipulation, deception, etc. While these capabilities can be attained separately, we expect that their development is correlated, e.g. all of them likely increase with scale.

There are various issues with the word "intelligence" that make it less suitable than "capabilities" for discussing risks from general AI systems:

Anthropomorphism: people often specifically associate "intelligence" with being human, being conscious, being alive, or having human-like emotions (none of which are relevant to or a prerequisite for risks posed by general AI systems).

Associations with harmful beliefs and ideologies.

Moving goalposts: impressive achievements in AI are often dismissed as not indicating "true intelligence" or "real understanding" (e.g. see the "stochastic parrots" argument). Catastrophic risk concerns are based on what the AI system can do, not whether it has "real understanding" of language or the world.

Stronger associations with less risky capabilities: people are more likely to associate "intelligence" with being really good at math than being really good at politics, while the latter may be more representative of capabilities that make general AI systems pose a risk (e.g. manipulation and deception capabilities that could enable the system to overpower humans).

High level of abstraction: "intelligence" can take on the quality of a mythical ideal that can't be met by an actual AI system, while "competence" is more conducive to being specific about the capability level in question.

It's worth noting that I am not suggesting to always avoid the term "intelligence" when discussing advanced AI systems. Those who are trying to build advanced AI systems often want to capture different aspects of intelligence or endow the system with real understanding of the world, and it's useful to investigate and discuss to what extent an AI system has (or could have) these properties. I am specifically advocating to avoid the term "intelligence" when discussing catastrophic risks, because AI systems can pose these risks without possessing real understanding or some particular aspects of intelligence.

The basic argument for catastrophic risk from general AI has two parts: 1) the world is on track to develop generally capable AI systems in the next few decades, and 2) generally capable AI systems are likely to outcompete or overpower humans. Both of these arguments ar...