Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ten arguments that AI is an existential risk, published by KatjaGrace on August 13, 2024 on LessWrong.

This is a snapshot of a new page on the AI Impacts Wiki.

We've made a list of arguments[1] that AI poses an existential risk to humanity. We'd love to hear how you feel about them in the comments and polls.

Competent non-aligned agents

Summary:

1. Humans will build AI systems that are 'agents', i.e. they will autonomously pursue goals

2. Humans won't figure out how to make systems with goals that are compatible with human welfare and realizing human values

3. Such systems will be built or selected to be highly competent, and so gain the power to achieve their goals

4. Thus the future will be primarily controlled by AIs, who will direct it in ways that are at odds with long-run human welfare or the realization of human values

Selected counterarguments:

It is unclear that AI will tend to have goals that are bad for humans

There are many forms of power. It is unclear that a competence advantage will ultimately trump all others in time

This argument also appears to apply to human groups such as corporations, so we need an explanation of why those are not an existential risk

People who have favorably discussed[2] this argument (specific quotes here): Paul Christiano (2021), Ajeya Cotra (2023), Eliezer Yudkowsky (2024), Nick Bostrom (2014[3]).

See also: Full wiki page on the competent non-aligned agents argument

Second species argument

Summary:

1. Human dominance over other animal species is primarily due to humans having superior cognitive and coordination abilities

2. Therefore if another 'species' appears with abilities superior to those of humans, that species will become dominant over humans in the same way

3. AI will essentially be a 'species' with superior abilities to humans

4. Therefore AI will dominate humans

Selected counterarguments:

Human dominance over other species is plausibly not due to the cognitive abilities of individual humans, but rather because of human ability to communicate and store information through culture and artifacts

Intelligence in animals doesn't appear to generally relate to dominance. For instance, elephants are much more intelligent than beetles, and it is not clear that elephants have dominated beetles

Differences in capabilities don't necessarily lead to extinction. In the modern world, more powerful countries arguably control less powerful countries, but they do not wipe them out and most colonized countries have eventually gained independence

People who have favorably discussed this argument (specific quotes here): Joe Carlsmith (2024), Richard Ngo (2020), Stuart Russell (2020[4]), Nick Bostrom (2015).

See also: Full wiki page on the second species argument

Loss of control via inferiority

Summary:

1. AI systems will become much more competent than humans at decision-making

2. Thus most decisions will probably be allocated to AI systems

3. If AI systems make most decisions, humans will lose control of the future

4. If humans have no control of the future, the future will probably be bad for humans

Selected counterarguments:

Humans do not generally seem to become disempowered by possession of software that is far superior to them, even if it makes many 'decisions' in the process of carrying out their will

In the same way that humans avoid being overpowered by companies, even though companies are more competent than individual humans, humans can track AI trustworthiness and have AI systems compete for them as users. This might substantially mitigate untrustworthy AI behavior

People who have favorably discussed this argument (specific quotes here): Paul Christiano (2014), Ajeya Cotra (2023), Richard Ngo (2024).

See also: Full wiki page on loss of control via inferiority

Loss of control via speed

Summary:

1. Advances in AI will produce...