Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Should we rely on the speed prior for safety?, published by Marc-Everin Carauleanu on December 14, 2021 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

TL;DR: There is some hope that by penalizing time complexity we could strongly select for non-deceptive models as they require time to figure out if they are out of training. Evan Hubinger argued in his post ‘Are minimal circuits deceptive?’ that we cannot fully rely on the speed prior to ensure non-deceptiveness by attempting to show that even minimal circuits, which we can intuitively understand as “fastest programs”, can be deceptive. This is because of the risk of spontaneous meta-learning which doesn’t inherit the safety guarantees that we would hope to get from the speed prior. This argument could be wrong if meta-learning is never the fastest way to solve a problem or if spontaneous meta-learning is constrained by the speed prior.

Introduction

A possible approach to tackling inner alignment failures is to shape the model’s inductive biases to make them unlikely. The two most natural inductive biases are the simplicity prior and the time complexity/speed prior. Increasing the number of parameters of the model is conducive to the simplicity prior as you are increasing the possible number of models from which SGD can optimize for the simplest model whereas decreasing the number of parameters is promoting the speed prior.[1] [2]

In ‘Risks from Learned Optimization’[3] the concept of deceptive alignment is presented. That is, a model is said to be deceptively aligned if during training it pretends to be aligned in order to defect later on once it’s deployed. Paul Christiano previously argued that in the limit of perfect capabilities the simplicity prior can result in deceptive models. That is because, between the models that are capable of deception, the ones with less complex (unaligned) objectives are simpler than the aligned ones which have a more complex objective. He then left the open question of whether penalizing time complexity can lead to a deceptive model or not.

Deceptive models have a particular property: they need to spend time figuring out if they are out of the training or not. The hope would be that by penalizing time complexity the model wouldn’t have time to figure out if it is out of training and therefore it would not be capable of deception.

Minimal circuits can be deceptive

Evan Hubinger attempted to resolve this open question in his post ‘Are minimal circuits deceptive?’ by thinking in terms of minimal circuits which we can intuitively understand as “fastest programs”. The conclusion of his argument is that even the fastest programs that solve a set of tasks can be deceptive, potentially crushing the hope that strongly penalizing time complexity can ensure non-deceptive models.

To understand why one has to first understand mesa-optimization. It is a neologism introduced in ‘Risks from Learned Optimization’ which means that a learned model is also an optimiser, in other words, if your optimizer (base-optimizer) learns another optimizer, then the learned optimizer is the mesa-optimizer.

Now consider the possibility that your learning algorithm learns a specific type of optimizer: another learning algorithm (meta-learning). There are some cases in which the fastest way to solve a problem (set of tasks) could be to do meta-learning, in fact, that’s what we do. It seems likely that for a problem like classifying objects needed for an average human world model, meta-learning is the fastest way to do that.

There is one issue with this though: the fastest meta-learning algorithms don’t necessarily learn the fastest models, which would eliminate the ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Should we rely on the speed prior for safety?, published by Marc-Everin Carauleanu on December 14, 2021 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

TL;DR: There is some hope that by penalizing time complexity we could strongly select for non-deceptive models as they require time to figure out if they are out of training. Evan Hubinger argued in his post ‘Are minimal circuits deceptive?’ that we cannot fully rely on the speed prior to ensure non-deceptiveness by attempting to show that even minimal circuits, which we can intuitively understand as “fastest programs”, can be deceptive. This is because of the risk of spontaneous meta-learning which doesn’t inherit the safety guarantees that we would hope to get from the speed prior. This argument could be wrong if meta-learning is never the fastest way to solve a problem or if spontaneous meta-learning is constrained by the speed prior.

Introduction

A possible approach to tackling inner alignment failures is to shape the model’s inductive biases to make them unlikely. The two most natural inductive biases are the simplicity prior and the time complexity/speed prior. Increasing the number of parameters of the model is conducive to the simplicity prior as you are increasing the possible number of models from which SGD can optimize for the simplest model whereas decreasing the number of parameters is promoting the speed prior.[1] [2]

In ‘Risks from Learned Optimization’[3] the concept of deceptive alignment is presented. That is, a model is said to be deceptively aligned if during training it pretends to be aligned in order to defect later on once it’s deployed. Paul Christiano previously argued that in the limit of perfect capabilities the simplicity prior can result in deceptive models. That is because, between the models that are capable of deception, the ones with less complex (unaligned) objectives are simpler than the aligned ones which have a more complex objective. He then left the open question of whether penalizing time complexity can lead to a deceptive model or not.

Deceptive models have a particular property: they need to spend time figuring out if they are out of the training or not. The hope would be that by penalizing time complexity the model wouldn’t have time to figure out if it is out of training and therefore it would not be capable of deception.

Minimal circuits can be deceptive

Evan Hubinger attempted to resolve this open question in his post ‘Are minimal circuits deceptive?’ by thinking in terms of minimal circuits which we can intuitively understand as “fastest programs”. The conclusion of his argument is that even the fastest programs that solve a set of tasks can be deceptive, potentially crushing the hope that strongly penalizing time complexity can ensure non-deceptive models.

To understand why one has to first understand mesa-optimization. It is a neologism introduced in ‘Risks from Learned Optimization’ which means that a learned model is also an optimiser, in other words, if your optimizer (base-optimizer) learns another optimizer, then the learned optimizer is the mesa-optimizer.

Now consider the possibility that your learning algorithm learns a specific type of optimizer: another learning algorithm (meta-learning). There are some cases in which the fastest way to solve a problem (set of tasks) could be to do meta-learning, in fact, that’s what we do. It seems likely that for a problem like classifying objects needed for an average human world model, meta-learning is the fastest way to do that.

There is one issue with this though: the fastest meta-learning algorithms don’t necessarily learn the fastest models, which would eliminate the ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consequentialism & corrigibility, published by Steven Byrnes on December 14, 2021 on LessWrong.

Background 1: Preferences-over-future-states (a.k.a. consequentialism) vs Preferences-over-trajectories

The post Coherent decisions imply consistent utilities (Eliezer Yudkowsky, 2017) explains how, if an agent has preferences over future states of the world, they should act like a utility-maximizer (with utility function defined over future states of the world). If they don’t act that way, they will be less effective at satisfying their own preferences; they would be “leaving money on the table” by their own reckoning. And there are externally-visible signs of agents being suboptimal in that sense; I'll go over an example in a second.

By contrast, the post Coherence arguments do not entail goal-directed behavior (Rohin Shah, 2018) notes that, if an agent has preferences over trajectories (a.k.a. universe-histories), and acts optimally with respect to those preferences (acts as a utility-maximizer whose utility function is defined over trajectories), then they can display any external behavior whatsoever. In other words, there's no externally-visible behavioral pattern which we can point to and say "That's a sure sign that this agent is behaving suboptimally, with respect to their own preferences.".

For example, the first (Yudkowsky) post mentions a hypothetical person at a restaurant. When they have an onion pizza, they’ll happily pay $0.01 to trade it for a pineapple pizza. When they have a pineapple pizza, they’ll happily pay $0.01 to trade it for a mushroom pizza. When they have a mushroom pizza, they’ll happily pay $0.01 to trade it for a pineapple pizza. The person goes around and around, wasting their money in a self-defeating way (a.k.a. “getting money-pumped”).

That post describes the person as behaving sub-optimally. But if you read carefully, the author sneaks in a critical background assumption: the person in question has preferences about what pizza they wind up eating, and they’re making these decisions based on those preferences. But what if they don’t? What if the person has no preference whatsoever about pizza? What if instead they’re an asshole restaurant customer who derives pure joy from making the waiter run back and forth to the kitchen?! Then we can look at the same behavior, and we wouldn’t describe it as self-defeating “getting money-pumped”, instead we would describe it as the skillful satisfaction of the person’s own preferences! They’re buying cheap entertainment! So that would be an example of preferences-over-trajectories.

To be more concrete, if I’m deciding between two possible courses of action, A and B, “preference over future states” would make the decision based on the state of the world after I finish the course of action—or more centrally, long after I finish the course of action. By contrast, “preference over trajectories” would allow the decision to depend on anything, even including what happens during the course-of-action.

(Strictly speaking, "preferences over trajectories" should be a superset of "preferences over future states", but in this post I'm treating them as exclusive—i.e., when I write "preferences over trajectories" in this post, I'm hiding an implicit ".other than those that could also be described as a preferences over future states".)

Background 2: Corrigibility is a square peg, preferences-over-future-states is a round hole

A “corrigible” AI is an AI for which you can shut it off (or more generally change its goals), and it doesn’t try to stop you. It also doesn’t deactivate its own shutoff switch, and it even fixes the switch if it breaks. Nor does it have preferences in the opposite direction: it doesn’t try to press the switch itself, and it doesn’t try to persuade you to press the switch. (Note: I’m using the te...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consequentialism & corrigibility, published by Steven Byrnes on December 14, 2021 on LessWrong.

Background 1: Preferences-over-future-states (a.k.a. consequentialism) vs Preferences-over-trajectories

The post Coherent decisions imply consistent utilities (Eliezer Yudkowsky, 2017) explains how, if an agent has preferences over future states of the world, they should act like a utility-maximizer (with utility function defined over future states of the world). If they don’t act that way, they will be less effective at satisfying their own preferences; they would be “leaving money on the table” by their own reckoning. And there are externally-visible signs of agents being suboptimal in that sense; I'll go over an example in a second.

By contrast, the post Coherence arguments do not entail goal-directed behavior (Rohin Shah, 2018) notes that, if an agent has preferences over trajectories (a.k.a. universe-histories), and acts optimally with respect to those preferences (acts as a utility-maximizer whose utility function is defined over trajectories), then they can display any external behavior whatsoever. In other words, there's no externally-visible behavioral pattern which we can point to and say "That's a sure sign that this agent is behaving suboptimally, with respect to their own preferences.".

For example, the first (Yudkowsky) post mentions a hypothetical person at a restaurant. When they have an onion pizza, they’ll happily pay $0.01 to trade it for a pineapple pizza. When they have a pineapple pizza, they’ll happily pay $0.01 to trade it for a mushroom pizza. When they have a mushroom pizza, they’ll happily pay $0.01 to trade it for a pineapple pizza. The person goes around and around, wasting their money in a self-defeating way (a.k.a. “getting money-pumped”).

That post describes the person as behaving sub-optimally. But if you read carefully, the author sneaks in a critical background assumption: the person in question has preferences about what pizza they wind up eating, and they’re making these decisions based on those preferences. But what if they don’t? What if the person has no preference whatsoever about pizza? What if instead they’re an asshole restaurant customer who derives pure joy from making the waiter run back and forth to the kitchen?! Then we can look at the same behavior, and we wouldn’t describe it as self-defeating “getting money-pumped”, instead we would describe it as the skillful satisfaction of the person’s own preferences! They’re buying cheap entertainment! So that would be an example of preferences-over-trajectories.

To be more concrete, if I’m deciding between two possible courses of action, A and B, “preference over future states” would make the decision based on the state of the world after I finish the course of action—or more centrally, long after I finish the course of action. By contrast, “preference over trajectories” would allow the decision to depend on anything, even including what happens during the course-of-action.

(Strictly speaking, "preferences over trajectories" should be a superset of "preferences over future states", but in this post I'm treating them as exclusive—i.e., when I write "preferences over trajectories" in this post, I'm hiding an implicit ".other than those that could also be described as a preferences over future states".)

Background 2: Corrigibility is a square peg, preferences-over-future-states is a round hole

A “corrigible” AI is an AI for which you can shut it off (or more generally change its goals), and it doesn’t try to stop you. It also doesn’t deactivate its own shutoff switch, and it even fixes the switch if it breaks. Nor does it have preferences in the opposite direction: it doesn’t try to press the switch itself, and it doesn’t try to persuade you to press the switch. (Note: I’m using the te...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ARC is hiring!, published by paulfchristiano on December 14, 2021 on LessWrong.

The Alignment Research Center is hiring researchers; if you are interested, please apply!

What is ARC?

ARC is a non-profit organization focused on theoretical research to align future machine learning systems with human interests. We are aiming to develop alignment strategies that would continue to work regardless of how far we scaled up ML or how ML models end up working internally.

Probably the best way to understand our work is to read Eliciting Latent Knowledge, a report describing some recent and upcoming research, which illustrates our general methodology.

We currently have 2 research staff (Paul Christiano and Mark Xu). We’re aiming to hire another 1-2 researchers in early 2022. ARC is a new organization and is hoping to grow significantly over the next few years, so early hires will play a key role in helping define and scale up our research.

Who should apply?

Most of all, you should send in an application if you feel excited about proposing the kinds of algorithms and counterexamples described in our report on ELK.

We’re open to anyone who is excited about working on alignment even if you don't yet have any research background or your current research is in another field. You may be an especially good fit if you:

Have thought a lot about the AI alignment problem, especially in the limit of very powerful AI systems.

Are creative and generative (e.g. you may already have some ideas for potential strategies or counterexamples for ELK, even if they don't work).

Have experience designing algorithms, proving theorems, or formalizing concepts.

Have a broad base of knowledge in mathematics and computer science (we often draw test cases and counterexamples from these fields).

We are based in Berkeley, CA and would prefer people who can work from our office, but we’re open to discussing remote arrangements for great candidates.

Hiring will be a priority for us in early 2022 and we don't mind reading a lot of applications, so feel free to err on the side of sending in an application.

The hiring process

You can apply by filling out this short form. We will begin reviewing applications and interviewing candidates on January 3, 2022. Our hiring process involves a series of 1-2 hours interviews followed by a paid day-long work sample. Where possible we also prefer to do a longer trial although we understand that’s not practical for everyone.

Q&A

We're particularly excited about answering questions posted here through the rest of December. We welcome any questions about what working at ARC is like, the hiring process, what we're looking for, or whether you should apply.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ARC is hiring!, published by paulfchristiano on December 14, 2021 on LessWrong.

The Alignment Research Center is hiring researchers; if you are interested, please apply!

What is ARC?

ARC is a non-profit organization focused on theoretical research to align future machine learning systems with human interests. We are aiming to develop alignment strategies that would continue to work regardless of how far we scaled up ML or how ML models end up working internally.

Probably the best way to understand our work is to read Eliciting Latent Knowledge, a report describing some recent and upcoming research, which illustrates our general methodology.

We currently have 2 research staff (Paul Christiano and Mark Xu). We’re aiming to hire another 1-2 researchers in early 2022. ARC is a new organization and is hoping to grow significantly over the next few years, so early hires will play a key role in helping define and scale up our research.

Who should apply?

Most of all, you should send in an application if you feel excited about proposing the kinds of algorithms and counterexamples described in our report on ELK.

We’re open to anyone who is excited about working on alignment even if you don't yet have any research background or your current research is in another field. You may be an especially good fit if you:

Have thought a lot about the AI alignment problem, especially in the limit of very powerful AI systems.

Are creative and generative (e.g. you may already have some ideas for potential strategies or counterexamples for ELK, even if they don't work).

Have experience designing algorithms, proving theorems, or formalizing concepts.

Have a broad base of knowledge in mathematics and computer science (we often draw test cases and counterexamples from these fields).

We are based in Berkeley, CA and would prefer people who can work from our office, but we’re open to discussing remote arrangements for great candidates.

Hiring will be a priority for us in early 2022 and we don't mind reading a lot of applications, so feel free to err on the side of sending in an application.

The hiring process

You can apply by filling out this short form. We will begin reviewing applications and interviewing candidates on January 3, 2022. Our hiring process involves a series of 1-2 hours interviews followed by a paid day-long work sample. Where possible we also prefer to do a longer trial although we understand that’s not practical for everyone.

Q&A

We're particularly excited about answering questions posted here through the rest of December. We welcome any questions about what working at ARC is like, the hiring process, what we're looking for, or whether you should apply.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ARC is hiring!, published by Paul Christiano on December 14, 2021 on The Effective Altruism Forum.

The Alignment Research Center is hiring researchers; if you are interested, please apply!

What is ARC?

ARC is a non-profit organization focused on theoretical research to align future machine learning systems with human interests. We are aiming to develop alignment strategies that would continue to work regardless of how far we scaled up ML or how ML models end up working internally.

Probably the best way to understand our work is to read Eliciting Latent Knowledge, a report describing some recent and upcoming research, which illustrates our general methodology.

We currently have 2 research staff (Paul Christiano and Mark Xu). We’re aiming to hire another 1-2 researchers in early 2022. ARC is a new organization and is hoping to grow significantly over the next few years, so early hires will play a key role in helping define and scale up our research.

Who should apply?

Most of all, you should send in an application if you feel excited about proposing the kinds of algorithms and counterexamples described in our report on ELK.

We’re open to anyone who is excited about working on alignment even if you don't yet have any research background or your current research is in another field. You may be an especially good fit if you:

Have thought a lot about the AI alignment problem, especially in the limit of very powerful AI systems.

Are creative and generative (e.g. you may already have some ideas for potential strategies or counterexamples for ELK, even if they don't work).

Have experience designing algorithms, proving theorems, or formalizing concepts.

Have a broad base of knowledge in mathematics and computer science (we often draw test cases and counterexamples from these fields).

We are based in Berkeley, CA and would prefer people who can work from our office, but we’re open to discussing remote arrangements for great candidates.

Hiring will be a priority for us in early 2022 and we don't mind reading a lot of applications, so feel free to err on the side of sending in an application.

The hiring process

You can apply by filling out this short form. We will begin reviewing applications and interviewing candidates on January 3, 2022. Our hiring process involves a series of 1-2 hours interviews followed by a paid day-long work sample. Where possible we also prefer to do a longer trial although we understand that’s not practical for everyone.

Q&A

We're particularly excited about answering questions posted here through the rest of December. We welcome any questions about what working at ARC is like, the hiring process, what we're looking for, or whether you should apply.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ARC is hiring!, published by Paul Christiano on December 14, 2021 on The Effective Altruism Forum.

The Alignment Research Center is hiring researchers; if you are interested, please apply!

What is ARC?

ARC is a non-profit organization focused on theoretical research to align future machine learning systems with human interests. We are aiming to develop alignment strategies that would continue to work regardless of how far we scaled up ML or how ML models end up working internally.

Probably the best way to understand our work is to read Eliciting Latent Knowledge, a report describing some recent and upcoming research, which illustrates our general methodology.

We currently have 2 research staff (Paul Christiano and Mark Xu). We’re aiming to hire another 1-2 researchers in early 2022. ARC is a new organization and is hoping to grow significantly over the next few years, so early hires will play a key role in helping define and scale up our research.

Who should apply?

Most of all, you should send in an application if you feel excited about proposing the kinds of algorithms and counterexamples described in our report on ELK.

We’re open to anyone who is excited about working on alignment even if you don't yet have any research background or your current research is in another field. You may be an especially good fit if you:

Have thought a lot about the AI alignment problem, especially in the limit of very powerful AI systems.

Are creative and generative (e.g. you may already have some ideas for potential strategies or counterexamples for ELK, even if they don't work).

Have experience designing algorithms, proving theorems, or formalizing concepts.

Have a broad base of knowledge in mathematics and computer science (we often draw test cases and counterexamples from these fields).

We are based in Berkeley, CA and would prefer people who can work from our office, but we’re open to discussing remote arrangements for great candidates.

Hiring will be a priority for us in early 2022 and we don't mind reading a lot of applications, so feel free to err on the side of sending in an application.

The hiring process

You can apply by filling out this short form. We will begin reviewing applications and interviewing candidates on January 3, 2022. Our hiring process involves a series of 1-2 hours interviews followed by a paid day-long work sample. Where possible we also prefer to do a longer trial although we understand that’s not practical for everyone.

Q&A

We're particularly excited about answering questions posted here through the rest of December. We welcome any questions about what working at ARC is like, the hiring process, what we're looking for, or whether you should apply.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ARC's first technical report: Eliciting Latent Knowledge, published by Paul Christiano on December 14, 2021 on The AI Alignment Forum.

ARC has published a report on Eliciting Latent Knowledge, an open problem which we believe is central to alignment. We think reading this report is the clearest way to understand what problems we are working on, how they fit into our plan for solving alignment in the worst case, and our research methodology.

The core difficulty we discuss is learning how to map between an AI’s model of the world and a human’s model. This is closely related to ontology identification (and other similar statements). Our main contribution is to present many possible approaches to the problem and a more precise discussion of why it seems to be difficult and important.

The report is available here as a google document. If you're excited about this research, we're hiring!

Q&A

We're particularly excited about answering questions posted here throughout December. We welcome any questions no matter how basic or confused; we would love to help people understand what research we’re doing and how we evaluate progress in enough detail that they could start to do it themselves.

Thanks to María Gutiérrez-Rojas for the illustrations in this piece (the good ones, blame us for the ugly diagrams). Thanks to Buck Shlegeris, Jon Uesato, Carl Shulman, and especially Holden Karnofsky for helpful discussions and comments.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ARC's first technical report: Eliciting Latent Knowledge, published by Paul Christiano on December 14, 2021 on The AI Alignment Forum.

ARC has published a report on Eliciting Latent Knowledge, an open problem which we believe is central to alignment. We think reading this report is the clearest way to understand what problems we are working on, how they fit into our plan for solving alignment in the worst case, and our research methodology.

The core difficulty we discuss is learning how to map between an AI’s model of the world and a human’s model. This is closely related to ontology identification (and other similar statements). Our main contribution is to present many possible approaches to the problem and a more precise discussion of why it seems to be difficult and important.

The report is available here as a google document. If you're excited about this research, we're hiring!

Q&A

We're particularly excited about answering questions posted here throughout December. We welcome any questions no matter how basic or confused; we would love to help people understand what research we’re doing and how we evaluate progress in enough detail that they could start to do it themselves.

Thanks to María Gutiérrez-Rojas for the illustrations in this piece (the good ones, blame us for the ugly diagrams). Thanks to Buck Shlegeris, Jon Uesato, Carl Shulman, and especially Holden Karnofsky for helpful discussions and comments.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.