This post was written by Mark Xu based on interviews with Carl Shulman. It was paid for by Open Philanthropy but is not representative of their views.

Summary

The absence of evidence for extraterrestrial life suggests the existence of a Great Filter, a set of factors that in combination drive the probability of a given star producing expansionist interstellar civilization very low.

The self-indication assumption (SIA) says that you should think it more probable that you exist in worlds where there are more observers faced with your exact observations.

Several SIA Doomsday Arguments note that there can be more naturally developed observers on primitive planets if the Great Filter lies after us, e.g. civilizations like ours are incapable of space travel, self-destruct, or are suppressed by hidden aliens; thus such arguments claim SIA gives overwhelming reason to expect a civilization like ours to fail to expand.

However, these so-called SIA Doomsday Arguments don’t take into account the possibility of mature civilizations simulating previous ones.

SIA overwhelmingly favors the hypothesis of simulations because it allows far more observers in our apparent position.

The number of such simulations is maximized when colonization is frequent enough for a large share of all resources to be colonized, but is largely indifferent about precise frequencies given that; so SIA does not suggest that unsimulated civilizations that look like Earth face a late Great Filter.

Note that we are not endorsing the underlying SIA decision making framework here, only discussing whether certain conclusions follow it. In dealing with such anthropic problems we would prefer approaches closer to Armstrong’s Anthropic decision theory, which we think is better for avoiding certain sorts of self-destructive anthropic confusions.

Introduction

The search for extraterrestrial intelligence has not yet yielded fruit. Hanson (1998) argues that this implies the probability of life evolving on a planet and becoming visible must be extremely low, a so-called Great Filter. Such a filter could be located in many possible difficulties: abiogenesis, intelligent life, interstellar colonization, etc. Humanity’s future prospects may depend on whether the difficulties have already passed or lie ahead. Thus we are left with a troubling question: how far along the filter are we?

Sandberg et al. (2018) observe that current scientific uncertainty is compatible with a high chance that we are alone in the universe. For example, Sandberg et al. suggest over 200 orders of magnitude of uncertainty over the frequency of abiogenesis. Since there is substantial prior probability on early Great Filters, the lack of visible extraterrestrial life can’t provide a very large likelihood ratio regarding late filters.

However, on the Self-Indication Assumption (SIA) the fact that we find ourselves to exist should provide overwhelming reason to reject theories of very large early filters, and purportedly favor late filters.

We will discuss how the Great Filter interacts with SIA. We will first introduce the assumption and then present Grace (2010) and Ord and Olsen (2020)’s arguments that SIA implies the Great Filter is ahead, which we will call the “SIA Doomsday Argument”. We will then argue that Bostrom’s simulation argument reverses the conclusion of the SIA Doomsday Argument.

Self-Indication Assumption

The self-indication assumption:

(SIA): Observers should reason as if they were a random sample from all possible observers. Observers should reason as if they have a probability of being in a world proportional with the number of observers it contains. Worlds where a higher number of observers are “like you” are more probable.

To illustrate applications of SIA, we will discuss two examples due to John Leslie and Nick Bostrom.

God’s Coin Toss: Suppose that God tosses a fair coin. If it comes up heads, he creates ten people, each in their own room. If tails, he creates one thousand people, eac...

This post was written by Mark Xu based on interviews with Carl Shulman. It was paid for by Open Philanthropy but is not representative of their views.

Summary

The absence of evidence for extraterrestrial life suggests the existence of a Great Filter, a set of factors that in combination drive the probability of a given star producing expansionist interstellar civilization very low.

The self-indication assumption (SIA) says that you should think it more probable that you exist in worlds where there are more observers faced with your exact observations.

Several SIA Doomsday Arguments note that there can be more naturally developed observers on primitive planets if the Great Filter lies after us, e.g. civilizations like ours are incapable of space travel, self-destruct, or are suppressed by hidden aliens; thus such arguments claim SIA gives overwhelming reason to expect a civilization like ours to fail to expand.

However, these so-called SIA Doomsday Arguments don’t take into account the possibility of mature civilizations simulating previous ones.

SIA overwhelmingly favors the hypothesis of simulations because it allows far more observers in our apparent position.

The number of such simulations is maximized when colonization is frequent enough for a large share of all resources to be colonized, but is largely indifferent about precise frequencies given that; so SIA does not suggest that unsimulated civilizations that look like Earth face a late Great Filter.

Note that we are not endorsing the underlying SIA decision making framework here, only discussing whether certain conclusions follow it. In dealing with such anthropic problems we would prefer approaches closer to Armstrong’s Anthropic decision theory, which we think is better for avoiding certain sorts of self-destructive anthropic confusions.

Introduction

The search for extraterrestrial intelligence has not yet yielded fruit. Hanson (1998) argues that this implies the probability of life evolving on a planet and becoming visible must be extremely low, a so-called Great Filter. Such a filter could be located in many possible difficulties: abiogenesis, intelligent life, interstellar colonization, etc. Humanity’s future prospects may depend on whether the difficulties have already passed or lie ahead. Thus we are left with a troubling question: how far along the filter are we?

Sandberg et al. (2018) observe that current scientific uncertainty is compatible with a high chance that we are alone in the universe. For example, Sandberg et al. suggest over 200 orders of magnitude of uncertainty over the frequency of abiogenesis. Since there is substantial prior probability on early Great Filters, the lack of visible extraterrestrial life can’t provide a very large likelihood ratio regarding late filters.

However, on the Self-Indication Assumption (SIA) the fact that we find ourselves to exist should provide overwhelming reason to reject theories of very large early filters, and purportedly favor late filters.

We will discuss how the Great Filter interacts with SIA. We will first introduce the assumption and then present Grace (2010) and Ord and Olsen (2020)’s arguments that SIA implies the Great Filter is ahead, which we will call the “SIA Doomsday Argument”. We will then argue that Bostrom’s simulation argument reverses the conclusion of the SIA Doomsday Argument.

Self-Indication Assumption

The self-indication assumption:

(SIA): Observers should reason as if they were a random sample from all possible observers. Observers should reason as if they have a probability of being in a world proportional with the number of observers it contains. Worlds where a higher number of observers are “like you” are more probable.

To illustrate applications of SIA, we will discuss two examples due to John Leslie and Nick Bostrom.

God’s Coin Toss: Suppose that God tosses a fair coin. If it comes up heads, he creates ten people, each in their own room. If tails, he creates one thousand people, eac...

Summary:

I saw some posts (here, here, here and here) discussing whether a neural net can "protect" a subnet from modification by gradient descent. I directly investigate this possibility using meta learning to find a neural network initialization that

Models the function f(x) = 1

Is very difficult to re-train with SGD to model another function (specifically f(x) = sin(x))

The meta learner is able to greatly reduce the efficacy of SGD during retraining by increasing the magnitude of the linear weight values (to over 100,000 in some cases). Edit: after closer examination (thanks evhub), it seems like the meta learner uses some other approach to prevent retraining. The increased weight magnitude was incidental. However, it can't protect the final layer's bias term from SGD, so instead the meta learner initially models f(x) = 1.68, so that it's close to modeling f(x) = 1 after the base optimizer reduces the final layer's bias term.

I've included my code and my meta trained model in this GitHub.

Background:

There are two functions: the base function and the meta function. The freshly initialized network should model the meta function. Then, the base optimizer will train the network to model the base function. My goal was to find a network initialization that continues to model the meta function, even after the base optimizer trains the network to model the base function.

It turns out that the process of gradient descent optimization is itself differentiable. Thus, we can start with a random network initialization, and repeat the following meta training cycle:

Train a freshly initialized network to model the base function.

Evaluate the trained network on the meta function.

Backpropegate from the loss on the meta function, through the training process, to the network initialization.

Methods:

I use the learn2learn framework to handle the meta learning. I train a 4 layer feed forward neural network (sizes 256, 128, 64, 64). Each cycle of training on the base function lasts between 70 and 120 training steps, with batch size 1024, a learning rate of 0.01, using the L1 loss between the output and f(x) = sin(x). The meta optimizer uses a learning rate of 0.1 and momentum of 0.92. Both base and meta optimizers use SGD. Additionally, I include the network’s initial performance on the meta objective (before any training on the base objective) in the meta loss function.

The meta training process was incredibly erratic, with losses spiking over into the hundreds of thousands before settling down. Later experiments showed a meta learning rate of ~0.01 helps control this.

Results:

For the randomly initialized network, we can see it quickly learns to model the base function f(x) = sin(x). In particular, training from the random initialization causes the outputs to begin bending to match the curves of sin(x) by step 10.

Summary:

I saw some posts (here, here, here and here) discussing whether a neural net can "protect" a subnet from modification by gradient descent. I directly investigate this possibility using meta learning to find a neural network initialization that

Models the function f(x) = 1

Is very difficult to re-train with SGD to model another function (specifically f(x) = sin(x))

The meta learner is able to greatly reduce the efficacy of SGD during retraining by increasing the magnitude of the linear weight values (to over 100,000 in some cases). Edit: after closer examination (thanks evhub), it seems like the meta learner uses some other approach to prevent retraining. The increased weight magnitude was incidental. However, it can't protect the final layer's bias term from SGD, so instead the meta learner initially models f(x) = 1.68, so that it's close to modeling f(x) = 1 after the base optimizer reduces the final layer's bias term.

I've included my code and my meta trained model in this GitHub.

Background:

There are two functions: the base function and the meta function. The freshly initialized network should model the meta function. Then, the base optimizer will train the network to model the base function. My goal was to find a network initialization that continues to model the meta function, even after the base optimizer trains the network to model the base function.

It turns out that the process of gradient descent optimization is itself differentiable. Thus, we can start with a random network initialization, and repeat the following meta training cycle:

Train a freshly initialized network to model the base function.

Evaluate the trained network on the meta function.

Backpropegate from the loss on the meta function, through the training process, to the network initialization.

Methods:

I use the learn2learn framework to handle the meta learning. I train a 4 layer feed forward neural network (sizes 256, 128, 64, 64). Each cycle of training on the base function lasts between 70 and 120 training steps, with batch size 1024, a learning rate of 0.01, using the L1 loss between the output and f(x) = sin(x). The meta optimizer uses a learning rate of 0.1 and momentum of 0.92. Both base and meta optimizers use SGD. Additionally, I include the network’s initial performance on the meta objective (before any training on the base objective) in the meta loss function.

The meta training process was incredibly erratic, with losses spiking over into the hundreds of thousands before settling down. Later experiments showed a meta learning rate of ~0.01 helps control this.

Results:

For the randomly initialized network, we can see it quickly learns to model the base function f(x) = sin(x). In particular, training from the random initialization causes the outputs to begin bending to match the curves of sin(x) by step 10.

This post contains some of my current best guesses at aspects of agent type signatures for which I expect there are useful Selection Theorems, as well as properties of selection optima which I expect are key to proving these type signatures.

This post assumes you have read the intro post on the Selection Theorem program. The intended audience is people who might work on the program, so these blurbs are intended to be link-heavy hooks and idea generators rather than self-contained explanations.

Aspects Of Agent Type Signature

These generally reflect my own current high-level thinking (though not all of the ideas are mine). For a different (and excellent, and better-written) take on similar questions from someone else, check out Mark and Evan’s Agents Over Cartesian World Models.

(A -> B) -> A

Scott Garrabrant’s post (A -> B) -> A provides the basic starting point for a Hofstadter-esque agent type signature. Breaking it down:

A represents the agent’s “actions”.

B represents the “outcomes” of those actions.

(A -> B) is the agent’s model of what outcomes result from an action. It’s a function which takes in actions, and spits out outcomes (that’s the arrow notation).

(A -> B) -> A is the full agent: it takes in the model (A -> B), and then outputs action A as a function of the model.

In pure abstract functional terms, the overall type signature is (A -> B) -> A. Lots of other type information is missing from this - there’s nothing about the types of the actions or the outcomes, for instance. But it seems like the right basic setup at a high level.

(A -> B) -> A + Abstraction

Generally speaking, low-level physical reality seems to be made of something like a giant causal network with symmetry. It doesn’t have any built-in “clouds” or “strange loops”. So, how do we get (A -> B) -> A structures in the physical world?

I expect that some chunks of the world naturally abstract into models with an agent-like type signature. In other words, approximately-all of the information about that chunk which propagates “far away” in the low-level causal network of the universe is summarized by a high-level model with an (A -> B) -> A signature.

This is the sort of “embedding” of agents for which I expect to find selection theorems.

World Models and Natural Abstractions

Humans’ world models sure do seem to have things in them - pencils, cars, other humans, etc. So a type signature for human world models (and, I expect, probably world models more generally) should give us some idea of what “things” are and how they correspond to structures in the world.

A different angle: to the extent that we expect to see roughly-Bayesian models (which is itself debatable, but at least better founded than EU maximization), what are the variables in those models, and how do they correspond to structures in the world?

This is another place where I expect the Natural Abstraction Hypothesis is relevant. Eventually, I expect to find selection theorems saying that agents evolve to reason using natural abstractions under a fairly broad range of conditions.

Goal Inputs

I’m surprised by just how much of a blindspot goal-inputs seem to be for today’s economists, AI researchers, etc. The coherence theorems usually cited to justify expected utility maximization models imply a quite narrow range of inputs to those utility functions: utilities are only over the outcomes on which agents can bet. Yet practitioners use utility functions over entire (unobservable) world states, world state trajectories, MDP states, etc, often without any way for the agent to bet on all of the outcomes.

Not that I think this is wrong; the inputs to human values seem to be abstract latent variables in humans’ world models, which are not things for which we can operationalize direct bets even in principle. (Indeed, this is arguably the fundamental barrier to making betting markets really useful.) The goal type signatures implied by the coherence theorems are not a good fit for realistic agenty syst...

This post contains some of my current best guesses at aspects of agent type signatures for which I expect there are useful Selection Theorems, as well as properties of selection optima which I expect are key to proving these type signatures.

This post assumes you have read the intro post on the Selection Theorem program. The intended audience is people who might work on the program, so these blurbs are intended to be link-heavy hooks and idea generators rather than self-contained explanations.

Aspects Of Agent Type Signature

These generally reflect my own current high-level thinking (though not all of the ideas are mine). For a different (and excellent, and better-written) take on similar questions from someone else, check out Mark and Evan’s Agents Over Cartesian World Models.

(A -> B) -> A

Scott Garrabrant’s post (A -> B) -> A provides the basic starting point for a Hofstadter-esque agent type signature. Breaking it down:

A represents the agent’s “actions”.

B represents the “outcomes” of those actions.

(A -> B) is the agent’s model of what outcomes result from an action. It’s a function which takes in actions, and spits out outcomes (that’s the arrow notation).

(A -> B) -> A is the full agent: it takes in the model (A -> B), and then outputs action A as a function of the model.

In pure abstract functional terms, the overall type signature is (A -> B) -> A. Lots of other type information is missing from this - there’s nothing about the types of the actions or the outcomes, for instance. But it seems like the right basic setup at a high level.

(A -> B) -> A + Abstraction

Generally speaking, low-level physical reality seems to be made of something like a giant causal network with symmetry. It doesn’t have any built-in “clouds” or “strange loops”. So, how do we get (A -> B) -> A structures in the physical world?

I expect that some chunks of the world naturally abstract into models with an agent-like type signature. In other words, approximately-all of the information about that chunk which propagates “far away” in the low-level causal network of the universe is summarized by a high-level model with an (A -> B) -> A signature.

This is the sort of “embedding” of agents for which I expect to find selection theorems.

World Models and Natural Abstractions

Humans’ world models sure do seem to have things in them - pencils, cars, other humans, etc. So a type signature for human world models (and, I expect, probably world models more generally) should give us some idea of what “things” are and how they correspond to structures in the world.

A different angle: to the extent that we expect to see roughly-Bayesian models (which is itself debatable, but at least better founded than EU maximization), what are the variables in those models, and how do they correspond to structures in the world?

This is another place where I expect the Natural Abstraction Hypothesis is relevant. Eventually, I expect to find selection theorems saying that agents evolve to reason using natural abstractions under a fairly broad range of conditions.

Goal Inputs

I’m surprised by just how much of a blindspot goal-inputs seem to be for today’s economists, AI researchers, etc. The coherence theorems usually cited to justify expected utility maximization models imply a quite narrow range of inputs to those utility functions: utilities are only over the outcomes on which agents can bet. Yet practitioners use utility functions over entire (unobservable) world states, world state trajectories, MDP states, etc, often without any way for the agent to bet on all of the outcomes.

Not that I think this is wrong; the inputs to human values seem to be abstract latent variables in humans’ world models, which are not things for which we can operationalize direct bets even in principle. (Indeed, this is arguably the fundamental barrier to making betting markets really useful.) The goal type signatures implied by the coherence theorems are not a good fit for realistic agenty syst...

Research projects

I'm planning to start two research projects on model splintering/reward generalisation and learning the preferences of irrational agents.

Within those projects, I'm aiming to work on subprojects that are:

Posed in terms that are familiar to conventional ML;

interesting to solve from the conventional ML perspective;

and whose solutions can be extended to the big issues in AI safety.

The point is not just to solve the sub-problems, but to solve them in ways that generalise or point to a general solution.

The aim is to iterate and improve fast on these ideas before implementing them. Because of that, these posts should be considered dynamic and prone to be re-edited, potentially often. Suggestions and modifications of the design are valuable and may get included in the top post.

AI learns promises and betrayal

Parent project: this is a subproject of both the model-splintering and the value learning projects.

Background

DeepMind's XLand allows the creation of multiple competitive and cooperative games. This paper shows the natural emergence of communication within cooperative multi-agent tasks. This paper shows that cooperation and communication can also emerge in first person shooter team games.

The idea is to combine these approaches to create multi-agent challenges where the agents learn to cooperate and communicate, and then mislead and betray each other. Ultimately the agents will learn the concepts of open and hidden betrayal. And we will attempt to get them to value avoiding committing those concepts.

Setup

As mentioned, the environment will be based on XLand, with mixed cooperative-competitive games, modified as needed to allow communication to develop between the agents.

We will attempt to create communication and cooperation between the agents, and then lead them to start betraying each other, and define the concept of betrayal.

Previous high level projects have tried to define concepts like "trustworthiness" (or the closely related "truthful") and motivated the AI to follow them. Here we will try the opposite: define "betrayal", and motivate the AIs to avoid it.

We'll try to categorise "open betrayal" (when the other players realise they are being betrayed) and "hidden betrayal" (where the other players don't realise it). It is the second one that is the most interesting, as avoiding hidden betrayal is closest to a moral value (a virtue, in fact). In contrast, avoiding open betrayal may have purely instrumental value, if other players are likely to retaliate or trust the agent less in the future.

We'll experiment with ways of motivating the agents to avoid betrayals, or getting anywhere near to them, and see if these ideas scale. We can make some agents artificially much more powerful and knowledgeable than others, giving us some experimental methods to check how performance changes with increased power.

Research aims

A minor research aim is to see how swiftly lying and betrayal can be generated in multi-player games, and what effect they have on all agents' overall scores.

A major aim is to have the agents identify a cluster of behaviours that correspond to "overt betrayal" and "secret betrayal".

We can then experiment with various ways of motivating the agents to avoid that behaviour; maybe a continuously rising penalty as they approach the boundary of the concept, to motivate them to stay well away.

Finally, we can see how an agent's behaviour scales as they become more powerful relative to the other agents.

This research will be more unguided than other subprojects; I'm not sure what we will find, or whether or not we will succeed.

The ideal would be if we can define "avoid secret betrayal" well enough that extending the behaviour becomes a problem of model splintering rather than "nearest unblocked strategy". Thus the agent will not do anything that is clearly a betrayal, or clearly close to a betrayal.

Research projects

I'm planning to start two research projects on model splintering/reward generalisation and learning the preferences of irrational agents.

Within those projects, I'm aiming to work on subprojects that are:

Posed in terms that are familiar to conventional ML;

interesting to solve from the conventional ML perspective;

and whose solutions can be extended to the big issues in AI safety.

The point is not just to solve the sub-problems, but to solve them in ways that generalise or point to a general solution.

The aim is to iterate and improve fast on these ideas before implementing them. Because of that, these posts should be considered dynamic and prone to be re-edited, potentially often. Suggestions and modifications of the design are valuable and may get included in the top post.

AI learns promises and betrayal

Parent project: this is a subproject of both the model-splintering and the value learning projects.

Background

DeepMind's XLand allows the creation of multiple competitive and cooperative games. This paper shows the natural emergence of communication within cooperative multi-agent tasks. This paper shows that cooperation and communication can also emerge in first person shooter team games.

The idea is to combine these approaches to create multi-agent challenges where the agents learn to cooperate and communicate, and then mislead and betray each other. Ultimately the agents will learn the concepts of open and hidden betrayal. And we will attempt to get them to value avoiding committing those concepts.

Setup

As mentioned, the environment will be based on XLand, with mixed cooperative-competitive games, modified as needed to allow communication to develop between the agents.

We will attempt to create communication and cooperation between the agents, and then lead them to start betraying each other, and define the concept of betrayal.

Previous high level projects have tried to define concepts like "trustworthiness" (or the closely related "truthful") and motivated the AI to follow them. Here we will try the opposite: define "betrayal", and motivate the AIs to avoid it.

We'll try to categorise "open betrayal" (when the other players realise they are being betrayed) and "hidden betrayal" (where the other players don't realise it). It is the second one that is the most interesting, as avoiding hidden betrayal is closest to a moral value (a virtue, in fact). In contrast, avoiding open betrayal may have purely instrumental value, if other players are likely to retaliate or trust the agent less in the future.

We'll experiment with ways of motivating the agents to avoid betrayals, or getting anywhere near to them, and see if these ideas scale. We can make some agents artificially much more powerful and knowledgeable than others, giving us some experimental methods to check how performance changes with increased power.

Research aims

A minor research aim is to see how swiftly lying and betrayal can be generated in multi-player games, and what effect they have on all agents' overall scores.

A major aim is to have the agents identify a cluster of behaviours that correspond to "overt betrayal" and "secret betrayal".

We can then experiment with various ways of motivating the agents to avoid that behaviour; maybe a continuously rising penalty as they approach the boundary of the concept, to motivate them to stay well away.

Finally, we can see how an agent's behaviour scales as they become more powerful relative to the other agents.

This research will be more unguided than other subprojects; I'm not sure what we will find, or whether or not we will succeed.

The ideal would be if we can define "avoid secret betrayal" well enough that extending the behaviour becomes a problem of model splintering rather than "nearest unblocked strategy". Thus the agent will not do anything that is clearly a betrayal, or clearly close to a betrayal.

This post is part 4 in our sequence on Modeling Transformative AI Risk. We are building a model to understand debates around existential risks from advanced AI. The model is made with Analytica software, and consists of nodes (representing key hypotheses and cruxes) and edges (representing the relationships between these cruxes), with final outputs corresponding to the likelihood of various potential failure scenarios. You can read more about the motivation for our project and how the model works in the Introduction post. The previous post in the sequence, Paths to High-Level Machine Intelligence, investigated how and when HLMI will be developed.

We are interested in feedback on this post, especially in places where the model does not capture your views or fails to include an uncertainty that you think could be an important crux. Similarly, if an explanation seems confused or confusing, flagging this is useful – both to help us clarify, and to ensure it doesn’t reflect an actual disagreement.

The goal of this part of the model is to describe the different potential characteristics of a transition from (pre-)HLMI¹ to superintelligent AI (i.e., “AI takeoff”). We also aim to clarify the relationships between these characteristics, and explain what assumptions they are sensitive to.

As shown in the above image, the relevant sections of the model (circled in light blue) take inputs primarily from these modules:

Analogies and General Priors on Intelligence – this module concerns both arguments that compare HLMI development with other, previous developments (for example, the evolution of human intelligence, current progress in machine learning, and past historical/cultural transitions) and broad philosophical claims about the nature of intelligence

High-Level Machine Intelligence (HLMI) – this module concerns whether HLMI will be developed at all, and if so, which type of HLMI and when

Hardware Overhang – this module concerns whether, at the time HLMI is developed, the amount of hardware required to run HLMI and the availability of such hardware will lead to a situation of “hardware overhang”, where overwhelming hardware resources allow for the first HLMI(s) to be overpowered due to being cheaply duplicated, easily run at a high clock speed, or otherwise rapidly scaled up.

The outputs of the sections of concern in this post, corresponding to the above circled modules, are:

Intelligence Explosion – will a positive feedback loop involving AI capabilities lead these capabilities to grow roughly hyperbolically across a sufficient range, such that the capabilities eventually grow incredibly quickly to an incredibly high level (presumably before plateauing as they approach some fundamental limit)?

Discontinuity around HLMI without self-improvement – will there be a rapid jump in AI capabilities from pre-HLMI AI to HLMI (for instance, if pre-HLMI acts like a machine with missing gears) and/or from HLMI to much higher intelligence (for instance, if a hardware overhang allows for rapid capability gain soon after HLMI)?

Takeoff Speed – how fast will the global economy (or the next closest thing, if this concept doesn’t transfer to a post-HLMI world) grow, once HLMI has matured as a technology?

HLMI is Distributed – will AI capabilities in a post-HLMI world be dispersed among many comparably powerful systems? A negative answer to this node indicates that HLMI capabilities will be concentrated in a few powerful systems (or a single such system).

These outputs provide a rough way of characterizing AI takeoff scenarios. While they are non-exhaustive, we believe they are a simple way of characterizing the range of outcomes which those who have seriously considered AI takeoff tend to find plausible. For instance, we can summarize (our understanding of) the publicly espoused views of Eliezer Yudkowsky, Paul Christiano, and Robin Hanson (along with a sceptic position) as follows:

(No significant intermediate doublings)

~hyperbolically increasing

(w...

This post is part 4 in our sequence on Modeling Transformative AI Risk. We are building a model to understand debates around existential risks from advanced AI. The model is made with Analytica software, and consists of nodes (representing key hypotheses and cruxes) and edges (representing the relationships between these cruxes), with final outputs corresponding to the likelihood of various potential failure scenarios. You can read more about the motivation for our project and how the model works in the Introduction post. The previous post in the sequence, Paths to High-Level Machine Intelligence, investigated how and when HLMI will be developed.

We are interested in feedback on this post, especially in places where the model does not capture your views or fails to include an uncertainty that you think could be an important crux. Similarly, if an explanation seems confused or confusing, flagging this is useful – both to help us clarify, and to ensure it doesn’t reflect an actual disagreement.

The goal of this part of the model is to describe the different potential characteristics of a transition from (pre-)HLMI¹ to superintelligent AI (i.e., “AI takeoff”). We also aim to clarify the relationships between these characteristics, and explain what assumptions they are sensitive to.

As shown in the above image, the relevant sections of the model (circled in light blue) take inputs primarily from these modules:

Analogies and General Priors on Intelligence – this module concerns both arguments that compare HLMI development with other, previous developments (for example, the evolution of human intelligence, current progress in machine learning, and past historical/cultural transitions) and broad philosophical claims about the nature of intelligence

High-Level Machine Intelligence (HLMI) – this module concerns whether HLMI will be developed at all, and if so, which type of HLMI and when

Hardware Overhang – this module concerns whether, at the time HLMI is developed, the amount of hardware required to run HLMI and the availability of such hardware will lead to a situation of “hardware overhang”, where overwhelming hardware resources allow for the first HLMI(s) to be overpowered due to being cheaply duplicated, easily run at a high clock speed, or otherwise rapidly scaled up.

The outputs of the sections of concern in this post, corresponding to the above circled modules, are:

Intelligence Explosion – will a positive feedback loop involving AI capabilities lead these capabilities to grow roughly hyperbolically across a sufficient range, such that the capabilities eventually grow incredibly quickly to an incredibly high level (presumably before plateauing as they approach some fundamental limit)?

Discontinuity around HLMI without self-improvement – will there be a rapid jump in AI capabilities from pre-HLMI AI to HLMI (for instance, if pre-HLMI acts like a machine with missing gears) and/or from HLMI to much higher intelligence (for instance, if a hardware overhang allows for rapid capability gain soon after HLMI)?

Takeoff Speed – how fast will the global economy (or the next closest thing, if this concept doesn’t transfer to a post-HLMI world) grow, once HLMI has matured as a technology?

HLMI is Distributed – will AI capabilities in a post-HLMI world be dispersed among many comparably powerful systems? A negative answer to this node indicates that HLMI capabilities will be concentrated in a few powerful systems (or a single such system).

These outputs provide a rough way of characterizing AI takeoff scenarios. While they are non-exhaustive, we believe they are a simple way of characterizing the range of outcomes which those who have seriously considered AI takeoff tend to find plausible. For instance, we can summarize (our understanding of) the publicly espoused views of Eliezer Yudkowsky, Paul Christiano, and Robin Hanson (along with a sceptic position) as follows:

(No significant intermediate doublings)

~hyperbolically increasing

(w...