Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding and controlling auto-induced distributional shift, published by LRudL on December 13, 2021 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Initiative ML Alignment Theory Scholars (MATS) program.

TL;DR: The input distribution of a supervised machine learning system is modeled as an unchanging thing, and even in reinforcement learning there are types of distributional shift we want to avoid. However, ML systems are part of the world, and often have incentives for changing the distribution of data they act on (for example, changing the set of users or the preferences of those users in an online content recommendation system such that the users become easier to satisfy). A paper by Krueger, Maharaj, and Leike explores such concerns, and especially whether ML systems end up acting on incentives for distributional shift. They invent simple scenarios and see what training setups lead to agents non-myopically act to shift the future distribution, and find that introducing meta-learning alone is enough to cause this, and that sometimes the choice of learning algorithm (e.g. policy gradient vs Q-learning) has perhaps surprising effects on whether or not such incentives are taken.

The paper proposes shuffling agents between environments as a solution, but this has its flaws. The topic relates to some concerns with embedded agents and especially myopia.

Introduction

A machine learning system receives inputs and produces outputs. When we train it, we want the system to become good at producing the right output given the right input.

Consider the wider context in which the model is used. If the model is deployed, then its outputs are generally used to do something in the world (and even if not deployed, they have some effect on the world: the ML engineers look at them, their calculation heats up the computer circuits a bit, and so on).

We can visualise the chain of influence like this (where the blue arrows denote causality):

How does the model decrease its loss? It's subject to the whims of the input distribution X and the loss function, so it is incentivised to somehow learn to do well, as measured by the loss, on inputs that are likely given the distribution (though perhaps deceptively, or otherwise in an unaligned, non-robust way).

However, the above diagram isn't really correct. Instead it's more like this:

The model is part of the world too. The agent's options include not just making changes to how it maps inputs to outputs, but also modifying its loss function to make it easier to attain its objective, as well as modifying the the inputs distribution through its impacts on the world (what we're concerned about in this post). We see that these concerns are part of a more generalised set of concerns that appears from agents being embedded in the world, but in this post we will only consider the consequences of the input distribution specifically being part of the world.

The input of the model is drawn from some part of the world – users of a news site, to use a running example in the paper – and the outputs impact the world (potentially including the input distribution). For example, the users of a news site with a content-recommendation system might leave or join the site on the basis of the recommendations that the model serves them.

From the perspective of the model trying to minimise its loss, this provides a new way to improve its performance. The input distribution is no longer an immutable law it has to take for granted, but rather something that the model can shape if it's clever enough.

Definitions

The most general thing happening here is that the input distribution can change – this is called distributional shift. However, we specifically care about dist...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding and controlling auto-induced distributional shift, published by LRudL on December 13, 2021 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Initiative ML Alignment Theory Scholars (MATS) program.

TL;DR: The input distribution of a supervised machine learning system is modeled as an unchanging thing, and even in reinforcement learning there are types of distributional shift we want to avoid. However, ML systems are part of the world, and often have incentives for changing the distribution of data they act on (for example, changing the set of users or the preferences of those users in an online content recommendation system such that the users become easier to satisfy). A paper by Krueger, Maharaj, and Leike explores such concerns, and especially whether ML systems end up acting on incentives for distributional shift. They invent simple scenarios and see what training setups lead to agents non-myopically act to shift the future distribution, and find that introducing meta-learning alone is enough to cause this, and that sometimes the choice of learning algorithm (e.g. policy gradient vs Q-learning) has perhaps surprising effects on whether or not such incentives are taken.

The paper proposes shuffling agents between environments as a solution, but this has its flaws. The topic relates to some concerns with embedded agents and especially myopia.

Introduction

A machine learning system receives inputs and produces outputs. When we train it, we want the system to become good at producing the right output given the right input.

Consider the wider context in which the model is used. If the model is deployed, then its outputs are generally used to do something in the world (and even if not deployed, they have some effect on the world: the ML engineers look at them, their calculation heats up the computer circuits a bit, and so on).

We can visualise the chain of influence like this (where the blue arrows denote causality):

How does the model decrease its loss? It's subject to the whims of the input distribution X and the loss function, so it is incentivised to somehow learn to do well, as measured by the loss, on inputs that are likely given the distribution (though perhaps deceptively, or otherwise in an unaligned, non-robust way).

However, the above diagram isn't really correct. Instead it's more like this:

The model is part of the world too. The agent's options include not just making changes to how it maps inputs to outputs, but also modifying its loss function to make it easier to attain its objective, as well as modifying the the inputs distribution through its impacts on the world (what we're concerned about in this post). We see that these concerns are part of a more generalised set of concerns that appears from agents being embedded in the world, but in this post we will only consider the consequences of the input distribution specifically being part of the world.

The input of the model is drawn from some part of the world – users of a news site, to use a running example in the paper – and the outputs impact the world (potentially including the input distribution). For example, the users of a news site with a content-recommendation system might leave or join the site on the basis of the recommendations that the model serves them.

From the perspective of the model trying to minimise its loss, this provides a new way to improve its performance. The input distribution is no longer an immutable law it has to take for granted, but rather something that the model can shape if it's clever enough.

Definitions

The most general thing happening here is that the input distribution can change – this is called distributional shift. However, we specifically care about dist...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Response to Recent Criticisms of Longtermism, published by Avital Balwit on December 13, 2021 on The Effective Altruism Forum.

This piece is a response to two recent essays by Phil Torres which are critical of longtermism. It does not presume that the reader is familiar with longtermism, and is thus not directed towards regular Forum readers who likely will already be familiar with this material.

Introduction

Recently, Phil Torres wrote two essays which are critical of longtermism. For the sake of brevity, this piece does not summarize them but assumes the reader has previously read them. My view is that Torres misportrays longtermism. So, in this essay I introduce longtermism, and then explain and respond to the main criticisms that Torres offers of it.

This is a long piece, so I encourage you to skip to the sections which most interest you. Here is the basic structure:

Brief explanation of longtermism

Contextualizing some quotes from the Torres essays

Climate change and the importance of disentangling disagreements about values, empirics, and terms

What longtermists mean by potential, and how it isn’t something mysterious or abstract

The difficult and often painful tradeoffs involved with handling non-existential catastrophes and prioritization

Limiting conditions and longtermism, essentially what the longtermism can — and cannot — be used to justify.

Whether transhumanism, space expansionism, and total utilitarianism are inherently part of longtermism (spoiler: they’re not, but there are some good reasons some longtermists support them)

Longtermism’s view (and lack of consensus) on technological development.

Conclusion (and mention of all the sections I had to cut).

What is longtermism?

Longtermism is the view that positively influencing the long term future is a key moral priority of our time. It’s based on the ideas that future people have moral worth, there could be very large numbers of future people, and that what we do today can affect how well or poorly their lives go [1].

Humanity might last for a very long time. A typical species’ lifespan would mean there are hundreds of thousands of years ahead of us [2] and the Earth will remain habitable for hundreds of millions of years [3]. If history were a novel, we may be living on its very first page [4]. More than just focusing on this mind bending scope, we can imagine — at least vaguely — the characters that might populate it: billions and billions of people: people who will feel the sun on their skin, fall in love, laugh at a joke, and experience all the other joys that life has to offer. Yet, our society pays relatively little attention to how our actions might affect people in the future.

Concern for future generations is not a new idea. Environmentalists have advocated for the interests of future people for many decades. Concern for future generations is enshrined in the Iroquois Nation’s constitution. John Adams, the second U.S. president, believed American institutions might last for thousands of years [5], while Ben Franklin bequeathed money to American cities under the provision that it could only be used centuries later.

That being said, there are several distinctive aspects of recent longtermist research and thinking, including the sheer timescales under consideration, the particular global problems that have been highlighted, and the consideration for the immense potential value of the future. Those engaged in longtermist research often look for events that will impact not just centuries, but potentially the whole future of civilisation — which might amount to millions or even billions of years. As for the global problems, a particular focus has been on existential risks: risks that threaten the destruction of humanity’s long-term potential [6]. Risks that have been highlighted by longtermist resea...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Response to Recent Criticisms of Longtermism, published by Avital Balwit on December 13, 2021 on The Effective Altruism Forum.

This piece is a response to two recent essays by Phil Torres which are critical of longtermism. It does not presume that the reader is familiar with longtermism, and is thus not directed towards regular Forum readers who likely will already be familiar with this material.

Introduction

Recently, Phil Torres wrote two essays which are critical of longtermism. For the sake of brevity, this piece does not summarize them but assumes the reader has previously read them. My view is that Torres misportrays longtermism. So, in this essay I introduce longtermism, and then explain and respond to the main criticisms that Torres offers of it.

This is a long piece, so I encourage you to skip to the sections which most interest you. Here is the basic structure:

Brief explanation of longtermism

Contextualizing some quotes from the Torres essays

Climate change and the importance of disentangling disagreements about values, empirics, and terms

What longtermists mean by potential, and how it isn’t something mysterious or abstract

The difficult and often painful tradeoffs involved with handling non-existential catastrophes and prioritization

Limiting conditions and longtermism, essentially what the longtermism can — and cannot — be used to justify.

Whether transhumanism, space expansionism, and total utilitarianism are inherently part of longtermism (spoiler: they’re not, but there are some good reasons some longtermists support them)

Longtermism’s view (and lack of consensus) on technological development.

Conclusion (and mention of all the sections I had to cut).

What is longtermism?

Longtermism is the view that positively influencing the long term future is a key moral priority of our time. It’s based on the ideas that future people have moral worth, there could be very large numbers of future people, and that what we do today can affect how well or poorly their lives go [1].

Humanity might last for a very long time. A typical species’ lifespan would mean there are hundreds of thousands of years ahead of us [2] and the Earth will remain habitable for hundreds of millions of years [3]. If history were a novel, we may be living on its very first page [4]. More than just focusing on this mind bending scope, we can imagine — at least vaguely — the characters that might populate it: billions and billions of people: people who will feel the sun on their skin, fall in love, laugh at a joke, and experience all the other joys that life has to offer. Yet, our society pays relatively little attention to how our actions might affect people in the future.

Concern for future generations is not a new idea. Environmentalists have advocated for the interests of future people for many decades. Concern for future generations is enshrined in the Iroquois Nation’s constitution. John Adams, the second U.S. president, believed American institutions might last for thousands of years [5], while Ben Franklin bequeathed money to American cities under the provision that it could only be used centuries later.

That being said, there are several distinctive aspects of recent longtermist research and thinking, including the sheer timescales under consideration, the particular global problems that have been highlighted, and the consideration for the immense potential value of the future. Those engaged in longtermist research often look for events that will impact not just centuries, but potentially the whole future of civilisation — which might amount to millions or even billions of years. As for the global problems, a particular focus has been on existential risks: risks that threaten the destruction of humanity’s long-term potential [6]. Risks that have been highlighted by longtermist resea...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is "gears-level" just a synonym for "mechanistic"?, published by capybaralet on December 13, 2021 on LessWrong.

If so, can we try to shift rationalist terminology towards the latter, which seems more transparent to outsiders?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is "gears-level" just a synonym for "mechanistic"?, published by capybaralet on December 13, 2021 on LessWrong.

If so, can we try to shift rationalist terminology towards the latter, which seems more transparent to outsiders?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What’s the backward-forward FLOP ratio for Neural Networks?, published by Marius Hobbhahn on December 13, 2021 on The AI Alignment Forum.

Summary:

Classic settings, i.e. deep networks with convolutional layers and large batch sizes, almost always have backward-forward FLOP ratios close to 2:1.

Depending on the following criteria we can encounter ratios between 1:1 and 3:1

Type of layer: Passes through linear layers have as many FLOP as they use to do weight updates. Convolutional layers have many more FLOP for passes than for weight updates. Therefore, in CNNs, FLOP for weight updates basically play no role.

Batch size: Weights are updated after the gradients of the batch have been aggregated. Thus, FLOP for passes increase with batch size but stay constant for weight updates.

Depth: The first layer has a backward-forward ratio of 1:1 while all others have 2:1. Therefore, the overall ratio is influenced by the fraction of FLOP in first vs. FLOP in other layers.

We assume the network is being optimized by stochastic gradient descent (w += ɑ⋅dw) and count the weight update as part of the backward pass. Other optimizers would imply different FLOP counts and could create ratios even larger than 3:1 for niche settings (see appendix B). However, the ratio of 2:1 in the classic setting (see point 1) should still hold even when you use momentum or Adam.

Compute-intensity of the weight update

Most compute-intensive layers

Backward-forward ratio

Large batch size OR compute-intensive convolutional layer

First layer

1:1

Other layers

2:1

Small batch size AND no compute-intensive convolutional layers

First layer

Other layers

3:1

Introduction:

As part of the Parameter, Compute and Data Trends in Machine Learning project we want to understand how many floating-point operations (FLOP) backward pass takes relative to a forward pass is for typical NN applications and which factors it depends on. We call this the backward-forward FLOP ratio.

In this post, we first provide a theoretical analysis of the ratio, and we then corroborate our findings empirically.

Theory:

To understand where the differences in ratios come from, we need to look at the classical equations of backpropagation.

Let’s start with a simple example---a neural network with 2 hidden layers.

In this example, we have the following computations for forward and backward pass assuming linear layers with ReLU activations. The “@”-symbols denote matrix multiplications.

Operation

Computation

FLOP forward

Computation

FLOP backward

δ1R=dL/dA2

=W2@δ2

dL/dW2

=δ2@A1R

We separate the weight update from the individual layers since the update is done after aggregation, i.e. we first add all gradients coming from different batches and then multiply with the learning rate.

From this table we see

ReLUs and the loss function contribute a negligible amount of FLOP compared to layers.

For the first layer, the backward-forward FLOP ratio is 1:1

For all other layers, the backward-forward FLOP ratio is 2:1 (ignoring ReLUs)

In equation form, the formula for the backward-forward FLOP ratio is:

backward / forward =

(FIRST LAYER FORWARD FLOP + 2OTHER LAYERS FORWARD FLOP + WEIGHT UPDATE) / (FIRST LAYER FORWARD FLOP + OTHER LAYERS FORWARD FLOP)

There are two considerations to see which terms dominate in this equation:

How much of the computation happens in the first layer?

How many operations does the weight update take compared to the computation in the layers? If the batch size is large or many parameters are shared, this term can be dismissed. Otherwise, it can be approximated as WEIGHT UPDATE ≈ FIRST LAYER FORWARD FLOP + OTHER LAYERS FORWARD FLOP.

This leads us to four possible cases:

Big weight update

Small weight update

First layer dominant

2FIRST LAYER FORWARD FLOP / FIRST LAYER FORWARD FLOP = 2:1

FIRST LAYER FORWARD FLOP / FIRST LAYER FORWARD ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What’s the backward-forward FLOP ratio for Neural Networks?, published by Marius Hobbhahn on December 13, 2021 on The AI Alignment Forum.

Summary:

Classic settings, i.e. deep networks with convolutional layers and large batch sizes, almost always have backward-forward FLOP ratios close to 2:1.

Depending on the following criteria we can encounter ratios between 1:1 and 3:1

Type of layer: Passes through linear layers have as many FLOP as they use to do weight updates. Convolutional layers have many more FLOP for passes than for weight updates. Therefore, in CNNs, FLOP for weight updates basically play no role.

Batch size: Weights are updated after the gradients of the batch have been aggregated. Thus, FLOP for passes increase with batch size but stay constant for weight updates.

Depth: The first layer has a backward-forward ratio of 1:1 while all others have 2:1. Therefore, the overall ratio is influenced by the fraction of FLOP in first vs. FLOP in other layers.

We assume the network is being optimized by stochastic gradient descent (w += ɑ⋅dw) and count the weight update as part of the backward pass. Other optimizers would imply different FLOP counts and could create ratios even larger than 3:1 for niche settings (see appendix B). However, the ratio of 2:1 in the classic setting (see point 1) should still hold even when you use momentum or Adam.

Compute-intensity of the weight update

Most compute-intensive layers

Backward-forward ratio

Large batch size OR compute-intensive convolutional layer

First layer

1:1

Other layers

2:1

Small batch size AND no compute-intensive convolutional layers

First layer

Other layers

3:1

Introduction:

As part of the Parameter, Compute and Data Trends in Machine Learning project we want to understand how many floating-point operations (FLOP) backward pass takes relative to a forward pass is for typical NN applications and which factors it depends on. We call this the backward-forward FLOP ratio.

In this post, we first provide a theoretical analysis of the ratio, and we then corroborate our findings empirically.

Theory:

To understand where the differences in ratios come from, we need to look at the classical equations of backpropagation.

Let’s start with a simple example---a neural network with 2 hidden layers.

In this example, we have the following computations for forward and backward pass assuming linear layers with ReLU activations. The “@”-symbols denote matrix multiplications.

Operation

Computation

FLOP forward

Computation

FLOP backward

δ1R=dL/dA2

=W2@δ2

dL/dW2

=δ2@A1R

We separate the weight update from the individual layers since the update is done after aggregation, i.e. we first add all gradients coming from different batches and then multiply with the learning rate.

From this table we see

ReLUs and the loss function contribute a negligible amount of FLOP compared to layers.

For the first layer, the backward-forward FLOP ratio is 1:1

For all other layers, the backward-forward FLOP ratio is 2:1 (ignoring ReLUs)

In equation form, the formula for the backward-forward FLOP ratio is:

backward / forward =

(FIRST LAYER FORWARD FLOP + 2OTHER LAYERS FORWARD FLOP + WEIGHT UPDATE) / (FIRST LAYER FORWARD FLOP + OTHER LAYERS FORWARD FLOP)

There are two considerations to see which terms dominate in this equation:

How much of the computation happens in the first layer?

How many operations does the weight update take compared to the computation in the layers? If the batch size is large or many parameters are shared, this term can be dismissed. Otherwise, it can be approximated as WEIGHT UPDATE ≈ FIRST LAYER FORWARD FLOP + OTHER LAYERS FORWARD FLOP.

This leads us to four possible cases:

Big weight update

Small weight update

First layer dominant

2FIRST LAYER FORWARD FLOP / FIRST LAYER FORWARD FLOP = 2:1

FIRST LAYER FORWARD FLOP / FIRST LAYER FORWARD ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Summary of the Acausal Attack Issue for AIXI, published by Diffractor on December 13, 2021 on The AI Alignment Forum.

Attention conservation notice: To a large extent, this is redundant with Paul's previous post about this, but I figured that some people might be interested in my restatement of the argument in my own words, as I did not start out believing that it was an issue, or start out agreeing with Vanessa about the attack operating via bridge rules.

Solomonoff induction/AIXI runs all possible computations, so it's possible in theory to alter which predictions a particular Turing machine outputs, by making stuff happen in your own universe, and this would then influence any process running that Turing machine, such as an AIXI agent. Of course, doing such a thing is subject to the obvious limitation where, if, say, you make the Turing machine that's reading your universe output a 0, you'll have less ability to influence the predictions and decisions of AIXI-like agents in the worlds where they see a 1 instead, because the Turing machine that you're controlling got eliminated for mispredictions.

Taking the attacker's perspective, if you were trying to influence one particular universe (this assumption will be loosened later) containing an AIXI or sufficiently AIXI-like target agent, and you had sufficiently high predictive abilities, you could try finding two low description complexity spots in your universe, one to check the state of, and one to write data to, and committing to the strategy "if the input data from this simple spot looks like the data I'd predict to receive from the world I'm interested in, I will submit output data accordingly (mostly accurate predictions of the victim's environment, but with whatever tweaks are needed), in order to influence what the target agent does in the targeted universe."

Basically, if you learn that there's an input channel to your universe, it's worthwhile to try to hack the output channel. It doesn't even take much effort to do, you just need to keep an eye on the conjectured input channel, commit to responding accordingly if it looks like it's transmitting data, and do other things in the meantime, and the Turing machine "run your universe, signal in via this channel, read output via this channel" is now under your control.

So... from the perspective of the targeted agent/universe, how well would your hacking attempt work? Well, it critically depends on whether the complexity of specifying "your universe + the I/O channels" is more than, or less than, the complexity of the shortest "honest" predictor of the observations of the targeted agent. If the honest predictor of the victim's observations is less complex than the specification of your universe and the I/O channels to it, then you messing around with the output channel and its predictions of observations would end up just affecting the 100th decimal place of the target AIXI's probabilities or something, because each bit is a factor of 2 difference in probability, and so you're at a huge disadvantage if you want to intentionally screw up the victim's probability estimates.

However, if the complexity of specifying your universe and the I/O channels is shorter than the "honest" predictor of the victim's observations, then after Solomonoff induction is done weeding out the mispredicting "chaff" hypotheses, the Turing machine that you're controlling is dominant over the "honest" predictor by an overwhelming factor (because each extra bit is a 2x difference in probability, so the situation is reversed). Now, just predict doom if the victim doesn't do what you want, and bam! You've taken control of the future of that target universe.

"But wait, how could specifying an entire universe containing an agent interested in hacking a target universe and competent enough to do ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Summary of the Acausal Attack Issue for AIXI, published by Diffractor on December 13, 2021 on The AI Alignment Forum.

Attention conservation notice: To a large extent, this is redundant with Paul's previous post about this, but I figured that some people might be interested in my restatement of the argument in my own words, as I did not start out believing that it was an issue, or start out agreeing with Vanessa about the attack operating via bridge rules.

Solomonoff induction/AIXI runs all possible computations, so it's possible in theory to alter which predictions a particular Turing machine outputs, by making stuff happen in your own universe, and this would then influence any process running that Turing machine, such as an AIXI agent. Of course, doing such a thing is subject to the obvious limitation where, if, say, you make the Turing machine that's reading your universe output a 0, you'll have less ability to influence the predictions and decisions of AIXI-like agents in the worlds where they see a 1 instead, because the Turing machine that you're controlling got eliminated for mispredictions.

Taking the attacker's perspective, if you were trying to influence one particular universe (this assumption will be loosened later) containing an AIXI or sufficiently AIXI-like target agent, and you had sufficiently high predictive abilities, you could try finding two low description complexity spots in your universe, one to check the state of, and one to write data to, and committing to the strategy "if the input data from this simple spot looks like the data I'd predict to receive from the world I'm interested in, I will submit output data accordingly (mostly accurate predictions of the victim's environment, but with whatever tweaks are needed), in order to influence what the target agent does in the targeted universe."

Basically, if you learn that there's an input channel to your universe, it's worthwhile to try to hack the output channel. It doesn't even take much effort to do, you just need to keep an eye on the conjectured input channel, commit to responding accordingly if it looks like it's transmitting data, and do other things in the meantime, and the Turing machine "run your universe, signal in via this channel, read output via this channel" is now under your control.

So... from the perspective of the targeted agent/universe, how well would your hacking attempt work? Well, it critically depends on whether the complexity of specifying "your universe + the I/O channels" is more than, or less than, the complexity of the shortest "honest" predictor of the observations of the targeted agent. If the honest predictor of the victim's observations is less complex than the specification of your universe and the I/O channels to it, then you messing around with the output channel and its predictions of observations would end up just affecting the 100th decimal place of the target AIXI's probabilities or something, because each bit is a factor of 2 difference in probability, and so you're at a huge disadvantage if you want to intentionally screw up the victim's probability estimates.

However, if the complexity of specifying your universe and the I/O channels is shorter than the "honest" predictor of the victim's observations, then after Solomonoff induction is done weeding out the mispredicting "chaff" hypotheses, the Turing machine that you're controlling is dominant over the "honest" predictor by an overwhelming factor (because each extra bit is a 2x difference in probability, so the situation is reversed). Now, just predict doom if the victim doesn't do what you want, and bam! You've taken control of the future of that target universe.

"But wait, how could specifying an entire universe containing an agent interested in hacking a target universe and competent enough to do ...