Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is: Eliciting latent knowledge: How to tell if your eyes deceive you, published by Paul Christiano, Ajeya Cotra, and Mark Xu on the AI Alignment Forum.

In this post, we’ll present ARC’s approach to an open problem we think is central to aligning powerful machine learning (ML) systems:

Suppose we train a model to predict what the future will look like according to cameras and other sensors. We then use planning algorithms to find a sequence of actions that lead to predicted futures that look good to us.

But some action sequences could tamper with the cameras so they show happy humans regardless of what’s really happening. More generally, some futures look great on camera but are actually catastrophically bad.

In these cases, the prediction model "knows" facts (like "the camera was tampered with") that are not visible on camera but would change our evaluation of the predicted future if we learned them. How can we train this model to report its latent knowledge of off-screen events?

We’ll call this problem eliciting latent knowledge (ELK). In this report we’ll focus on detecting sensor tampering as a motivating example, but we believe ELK is central to many aspects of alignment.

In this report, we will describe ELK and suggest possible approaches to it, while using the discussion to illustrate ARC’s research methodology. More specifically, we will:

Set up a toy scenario in which a prediction model could show us a future that looks good but is actually bad, and explain why ELK could address this problem (more).

Describe a simple baseline training strategy for ELK, step through how we analyze this kind of strategy, and ultimately conclude that the baseline is insufficient (more).

Lay out ARC’s overall research methodology — playing a game between a “builder” who is trying to come up with a good training strategy and a “breaker” who is trying to construct a counterexample where the strategy works poorly (more).

Describe a sequence of strategies for constructing richer datasets and arguments that none of these modifications solve ELK, leading to the counterexample of ontology identification (more).

Identify ontology identification as a crucial sub-problem of ELK and discuss its relationship to the rest of ELK (more).

Describe a sequence of strategies for regularizing models to give honest answers, and arguments that these modifications are still insufficient (more).

Conclude with a discussion of why we are excited about trying to solve ELK in the worst case, including why it seems central to the larger alignment problem and why we’re optimistic about making progress (more).

Much of our current research focused on “ontology identification” as a challenge for ELK. In the last 10 years many researchers have called out similar problems as playing a central role in alignment; our main contributions are to provide a more precise discussion of the problem, possible approaches, and why it appears to be challenging. We discuss related work in more detail in Appendix: related work.

We believe that there are many promising and unexplored approaches to this problem, and there isn’t yet much reason to believe we are stuck or are faced with an insurmountable obstacle. Even some of the simplest approaches have not been thoroughly explored, and seem like they would play a role in a practical attempt at scalable alignment today.

Given that ELK appears to represent a core difficulty for alignment, we are very excited about research that tries to attack it head on; we’re optimistic that within a year we will have made significant progress either towards a solution or towards a clear sense of why the problem is hard. If you’re interested in working with us on ELK or similar problems, get in touch.

Thanks to María Gutiérrez-Rojas for the illustrations in this piece. Thanks to Buck Shlegeris, Jon Ue...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is: Eliciting latent knowledge: How to tell if your eyes deceive you, published by Paul Christiano, Ajeya Cotra, and Mark Xu on the AI Alignment Forum.

In this post, we’ll present ARC’s approach to an open problem we think is central to aligning powerful machine learning (ML) systems:

Suppose we train a model to predict what the future will look like according to cameras and other sensors. We then use planning algorithms to find a sequence of actions that lead to predicted futures that look good to us.

But some action sequences could tamper with the cameras so they show happy humans regardless of what’s really happening. More generally, some futures look great on camera but are actually catastrophically bad.

In these cases, the prediction model "knows" facts (like "the camera was tampered with") that are not visible on camera but would change our evaluation of the predicted future if we learned them. How can we train this model to report its latent knowledge of off-screen events?

We’ll call this problem eliciting latent knowledge (ELK). In this report we’ll focus on detecting sensor tampering as a motivating example, but we believe ELK is central to many aspects of alignment.

In this report, we will describe ELK and suggest possible approaches to it, while using the discussion to illustrate ARC’s research methodology. More specifically, we will:

Set up a toy scenario in which a prediction model could show us a future that looks good but is actually bad, and explain why ELK could address this problem (more).

Describe a simple baseline training strategy for ELK, step through how we analyze this kind of strategy, and ultimately conclude that the baseline is insufficient (more).

Lay out ARC’s overall research methodology — playing a game between a “builder” who is trying to come up with a good training strategy and a “breaker” who is trying to construct a counterexample where the strategy works poorly (more).

Describe a sequence of strategies for constructing richer datasets and arguments that none of these modifications solve ELK, leading to the counterexample of ontology identification (more).

Identify ontology identification as a crucial sub-problem of ELK and discuss its relationship to the rest of ELK (more).

Describe a sequence of strategies for regularizing models to give honest answers, and arguments that these modifications are still insufficient (more).

Conclude with a discussion of why we are excited about trying to solve ELK in the worst case, including why it seems central to the larger alignment problem and why we’re optimistic about making progress (more).

Much of our current research focused on “ontology identification” as a challenge for ELK. In the last 10 years many researchers have called out similar problems as playing a central role in alignment; our main contributions are to provide a more precise discussion of the problem, possible approaches, and why it appears to be challenging. We discuss related work in more detail in Appendix: related work.

We believe that there are many promising and unexplored approaches to this problem, and there isn’t yet much reason to believe we are stuck or are faced with an insurmountable obstacle. Even some of the simplest approaches have not been thoroughly explored, and seem like they would play a role in a practical attempt at scalable alignment today.

Given that ELK appears to represent a core difficulty for alignment, we are very excited about research that tries to attack it head on; we’re optimistic that within a year we will have made significant progress either towards a solution or towards a clear sense of why the problem is hard. If you’re interested in working with us on ELK or similar problems, get in touch.

Thanks to María Gutiérrez-Rojas for the illustrations in this piece. Thanks to Buck Shlegeris, Jon Ue...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Opportunity to start a high-impact nonprofit - applications for the 2022-23 Charity Entrepreneurship Incubation Programs are now open!, published by KarolinaSarek on December 16, 2021 on The Effective Altruism Forum.

You can now apply to the two Incubation Programs 2022-23!

Charity Entrepreneurship’s Summer 2022 Incubation Program will be held online from late June to late August, and will include one week in person in London. The Winter 2023 program will be held from early February to late March, and will also include one week in person in the US (location to be determined).

Successful applicants will get a free 2-month intensive training in charity management, fundraising, impact analysis, and everything else you need to launch your own organization. Finish the program with a co-founder, a high-impact charity start-up, ongoing mentorship and support, and a funding proposal for a seed grant of up to $175,000. The deadline for first round applications is January 16, 2022.

Apply to start one of our new charity ideas from any country using the form below:

[APPLICATION FORM]

If you would like to apply with your own charity idea, please use this form:

[apply with your own idea]

Application process

Application deadline: January 16, 2022

Program date: 27 June – 19 August 2022, OR early February – late March 2023.

Costs: All costs covered.

The application process comprises five stages. Our entire application process typically takes less than 10 hours over approximately one month to complete. We’ve designed our vetting system to not only assess an applicant’s fit for CE, but also to help potential candidates get a sense of their other options and paths to impact. Due to the high number of applications, we are only able to contact applicants who are moving on to the second stage of the recruitment process.

Successful applicants will be informed whether they have been selected within one month of the application deadline. As we expect to accept around 10-20 candidates per program, we encourage you to apply early, though we will offer multiple application rounds.

We encourage any who may be interested in charity entrepreneurship to apply. We select attendees on the basis of potential, not track record alone. As we only accept applicants with a strong chance, given training, of starting a high-impact charity, the application process itself is the best indicator of possible fit. We also recommend looking over resources like our past blog content, videos, and reading list to learn more and/or prepare for the program.

The program

Our Incubation Program requires a full-time commitment (on average, 8 hours a day) during the two-month training course. Stipends are provided to those in need, and transportation costs are covered for those who can travel for the week in person. We also include a preparation stage in the weeks before the program begins in order to level the playing field and prepare attendees to hit the ground running.

Overview:

First application round: 15 December 2021 – 16 January 2022Second application round: Begins in March 2022Pre-program: Starting in May, you’ll read the Charity Entrepreneurship handbook (soon to be available on Amazon) and attend weekly one-hour virtual meetups to discuss content, ask questions, and get to know your potential co-founders. You will also take a test on pre-program content in June, before the program begins. Online program: 27 June – late19 August 2022, OR early February – late March 2023.First five weeks: You will work closely with other participants on real-world partner projects to be evaluated and scored by the CE Team. This will help you learn more about the intervention opportunities, choose a co-founder, and begin to build your charity. Final three weeks: Together with your co-founder, you will create a funding proposal to comp...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Opportunity to start a high-impact nonprofit - applications for the 2022-23 Charity Entrepreneurship Incubation Programs are now open!, published by KarolinaSarek on December 16, 2021 on The Effective Altruism Forum.

You can now apply to the two Incubation Programs 2022-23!

Charity Entrepreneurship’s Summer 2022 Incubation Program will be held online from late June to late August, and will include one week in person in London. The Winter 2023 program will be held from early February to late March, and will also include one week in person in the US (location to be determined).

Successful applicants will get a free 2-month intensive training in charity management, fundraising, impact analysis, and everything else you need to launch your own organization. Finish the program with a co-founder, a high-impact charity start-up, ongoing mentorship and support, and a funding proposal for a seed grant of up to $175,000. The deadline for first round applications is January 16, 2022.

Apply to start one of our new charity ideas from any country using the form below:

[APPLICATION FORM]

If you would like to apply with your own charity idea, please use this form:

[apply with your own idea]

Application process

Application deadline: January 16, 2022

Program date: 27 June – 19 August 2022, OR early February – late March 2023.

Costs: All costs covered.

The application process comprises five stages. Our entire application process typically takes less than 10 hours over approximately one month to complete. We’ve designed our vetting system to not only assess an applicant’s fit for CE, but also to help potential candidates get a sense of their other options and paths to impact. Due to the high number of applications, we are only able to contact applicants who are moving on to the second stage of the recruitment process.

Successful applicants will be informed whether they have been selected within one month of the application deadline. As we expect to accept around 10-20 candidates per program, we encourage you to apply early, though we will offer multiple application rounds.

We encourage any who may be interested in charity entrepreneurship to apply. We select attendees on the basis of potential, not track record alone. As we only accept applicants with a strong chance, given training, of starting a high-impact charity, the application process itself is the best indicator of possible fit. We also recommend looking over resources like our past blog content, videos, and reading list to learn more and/or prepare for the program.

The program

Our Incubation Program requires a full-time commitment (on average, 8 hours a day) during the two-month training course. Stipends are provided to those in need, and transportation costs are covered for those who can travel for the week in person. We also include a preparation stage in the weeks before the program begins in order to level the playing field and prepare attendees to hit the ground running.

Overview:

First application round: 15 December 2021 – 16 January 2022Second application round: Begins in March 2022Pre-program: Starting in May, you’ll read the Charity Entrepreneurship handbook (soon to be available on Amazon) and attend weekly one-hour virtual meetups to discuss content, ask questions, and get to know your potential co-founders. You will also take a test on pre-program content in June, before the program begins. Online program: 27 June – late19 August 2022, OR early February – late March 2023.First five weeks: You will work closely with other participants on real-world partner projects to be evaluated and scored by the CE Team. This will help you learn more about the intervention opportunities, choose a co-founder, and begin to build your charity. Final three weeks: Together with your co-founder, you will create a funding proposal to comp...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We'll Always Have Crazy, published by Duncan Sabien on December 15, 2021 on LessWrong.

Content warning: preparadigmatic meander; half-baked models; thinking-out-loud. Unlike the median Duncan essay.

Logan Strohl's response to the first draft of the below:

i really wish this were a more-effective-for-me invitation to help you figure something out. i think that's the version of it i would upvote, or maybe even strong upvote.

right now i feel uncomfortable or something about having read it, and if it were a topic i already cared a lot more about i'd probably feel angry or maybe even betrayed. as it is i feel a tiny bit like i've been contaminated, because of something like a wishy washy back and forth between trying to figure out what you're saying, trying to figure out what you're doing, having a feeling that there's something you want me to believe, trying to figure out what i do believe, trying to figure out what topics i need to try to get into my own sight before trying to understand you so that my own eyeballs don't get carelessly knocked over... whatever i'm trying to say here is more concentrated in that last thing

i think i would like a version of this post that begins with a short list of questions that i'm invited to try to answer on my own before i read, 'cause then i'd try to answer them on my own first and be much much better prepared for this essay

Do you think that the swath of society you live in is less crazy than society as a whole? Why?

Do you think that the swath of society you live in is less crazy than its analogue in society 100 years ago? Why?

How do you feel about the swaths of society that seem crazy to you? Why?

How do you feel about the future of society, on the question of craziness? Do you think the median craziness is moving? Do you think the extremes of craziness are moving? Do you think that the bulk of people are moving toward the middle, or the tails are fattening, or society is splitting into a two-peaked distribution, or or or or or? (various permutations of that kind of question)

What have you been using the word 'crazy' to mean, in the above? What do you think I use the word 'crazy' to mean? What do you think the median LWer uses it to mean?

What do you wish for, if anything, re: society and craziness?

For a long time, I didn't understand why elections were so often so close.

It seemed to kid-me that one side was (probably) strictly superior to the other, in the sense that the product of [the goodness of its goals] and [its effectiveness in pursuing those goals] was just straightforwardly higher.

And so it seemed to kid-me that once that side won once, it should be able to pretty quickly consolidate power and start accumulating compound gains from its achievements, at which point it should be wildly popular and win handily for a long time, until the underlying world state became Different enough that some new person or perspective would come out of nowhere and upend the existing order of things.

(I read a lot of fantasy books as a child.)

It took me a while to catch on to the fact that, actually, that exact process was happening. It's just that it was happening continuously rather than stepwise—subtle shifts, little swings, a constant string of tiny revolutions instead of rare, climactic battles.

It took me a while to realize that there's a sort of thermostat on political platforms. A party that wins in a landslide (or looks like it's on track to do so) will tend to fragment, as those within it become less-convinced that they need to put their important disagreements on hold for the greater good of Defeating The Other Guys, and more eager to fight the leadership (or, more cynically, to distinguish themselves from the crowd as someone with a reputation for Insight and Integrity).

And a party that suffers a major defeat softens and ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We'll Always Have Crazy, published by Duncan Sabien on December 15, 2021 on LessWrong.

Content warning: preparadigmatic meander; half-baked models; thinking-out-loud. Unlike the median Duncan essay.

Logan Strohl's response to the first draft of the below:

i really wish this were a more-effective-for-me invitation to help you figure something out. i think that's the version of it i would upvote, or maybe even strong upvote.

right now i feel uncomfortable or something about having read it, and if it were a topic i already cared a lot more about i'd probably feel angry or maybe even betrayed. as it is i feel a tiny bit like i've been contaminated, because of something like a wishy washy back and forth between trying to figure out what you're saying, trying to figure out what you're doing, having a feeling that there's something you want me to believe, trying to figure out what i do believe, trying to figure out what topics i need to try to get into my own sight before trying to understand you so that my own eyeballs don't get carelessly knocked over... whatever i'm trying to say here is more concentrated in that last thing

i think i would like a version of this post that begins with a short list of questions that i'm invited to try to answer on my own before i read, 'cause then i'd try to answer them on my own first and be much much better prepared for this essay

Do you think that the swath of society you live in is less crazy than society as a whole? Why?

Do you think that the swath of society you live in is less crazy than its analogue in society 100 years ago? Why?

How do you feel about the swaths of society that seem crazy to you? Why?

How do you feel about the future of society, on the question of craziness? Do you think the median craziness is moving? Do you think the extremes of craziness are moving? Do you think that the bulk of people are moving toward the middle, or the tails are fattening, or society is splitting into a two-peaked distribution, or or or or or? (various permutations of that kind of question)

What have you been using the word 'crazy' to mean, in the above? What do you think I use the word 'crazy' to mean? What do you think the median LWer uses it to mean?

What do you wish for, if anything, re: society and craziness?

For a long time, I didn't understand why elections were so often so close.

It seemed to kid-me that one side was (probably) strictly superior to the other, in the sense that the product of [the goodness of its goals] and [its effectiveness in pursuing those goals] was just straightforwardly higher.

And so it seemed to kid-me that once that side won once, it should be able to pretty quickly consolidate power and start accumulating compound gains from its achievements, at which point it should be wildly popular and win handily for a long time, until the underlying world state became Different enough that some new person or perspective would come out of nowhere and upend the existing order of things.

(I read a lot of fantasy books as a child.)

It took me a while to catch on to the fact that, actually, that exact process was happening. It's just that it was happening continuously rather than stepwise—subtle shifts, little swings, a constant string of tiny revolutions instead of rare, climactic battles.

It took me a while to realize that there's a sort of thermostat on political platforms. A party that wins in a landslide (or looks like it's on track to do so) will tend to fragment, as those within it become less-convinced that they need to put their important disagreements on hold for the greater good of Defeating The Other Guys, and more eager to fight the leadership (or, more cynically, to distinguish themselves from the crowd as someone with a reputation for Insight and Integrity).

And a party that suffers a major defeat softens and ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Universality and the “Filter”, published by Maggie Hayes on December 16, 2021 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

TL;DR: In this post, I refine universality, in order to explain the "Filter". The Filter is a method of risk analysis for the outputs of powerful AI systems. I refine universality by replacing the language of “beliefs” and proposing a handful of general definitions in part I. Then, I explain the key application of universality of building the Filter. I explain the Filter non-technically through a parable in part II, then technically in part III. To understand the Filter as presented in part II, it is not necessary to comprehend universality to the rigor that I present in part I; it is only necessary to see that all models are wrong1. While this post builds technically off a previous post2, part II is meant to be understandable to all audiences.

I. Refining Universality

Universality is discussed here, here, and here. Here are some features of universality that have been missing from its presentation.

1. The need to replace “beliefs”

A “belief” of a system is used 60 times in Paul Christiano’s attempt to formalize universality. I think a “belief” is not the right language for an object of a system for two reasons. First, it conflates terms worth making distinct. When “beliefs” are used in Paul Christiano’s attempt to formalize universality, they don’t always refer to the same thing. Second, “belief” should be reserved for a different concept. The way humans think of “belief” is not how it’s used there.

So, I rewrote the formalization of universality to not include the word “belief”. The handful of terms presented below combine to replace the previous notion of “belief”. Presenting universality without the language of “beliefs”, all said, is certainly more cumbersome.

I hope this reframing will help improve the methodology for applying universality to HCH. I apply the general definitions below to universality and the Filter in section III.2.

2. General definitions for universality

Consider κ, a list of action-shaping intelligible information. κ={κj}, where κj are pieces of intelligible information. Pieces of intelligible information are facts, which are sentences (i.e. “The sky is blue”, “there are infinitely many twin primes”). Another term, X, is the output of a computation. X is a list of actions, X={xl} (i.e. Printed to my computer the sentence “the sky is blue”, wiped the memory of Alice’s computer next door). A view may be modeled One may view a computation A as, upon an input of the world γ, taking a set of actions X. A(γ)=X. κ is a part of γ; action-shaping intelligible information of a system is part of the input of the world which determines the actions the system will take.

More formally:

Now, what was previously known as “belief” is now usually denoted as an action-shaping list of intelligible information, κ={κj}Jj=j0. In item II (above), a formal constraint of causality requires some causal theory, and also importing some causal assumptions. I do not provide this here. A causal constraint is necessary in addition to the statement A(κ)=xl to preclude the inclusion of arbitrary information in κ which are not related to the actions xl and which, without this constraint, are permitted to be very numerous.

I expect that item III (above) is closer to a common rule of thumb than to a universal truth. Some theories of causality between κ and xl might afford that in special circumstances, κ is unique (N=1).

Then, I may present universality with a couple formulas:

Note that the first and third statements are directly transcribed from Paul Christiano’s attempt to formalize universality.

The re...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Universality and the “Filter”, published by Maggie Hayes on December 16, 2021 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

TL;DR: In this post, I refine universality, in order to explain the "Filter". The Filter is a method of risk analysis for the outputs of powerful AI systems. I refine universality by replacing the language of “beliefs” and proposing a handful of general definitions in part I. Then, I explain the key application of universality of building the Filter. I explain the Filter non-technically through a parable in part II, then technically in part III. To understand the Filter as presented in part II, it is not necessary to comprehend universality to the rigor that I present in part I; it is only necessary to see that all models are wrong1. While this post builds technically off a previous post2, part II is meant to be understandable to all audiences.

I. Refining Universality

Universality is discussed here, here, and here. Here are some features of universality that have been missing from its presentation.

1. The need to replace “beliefs”

A “belief” of a system is used 60 times in Paul Christiano’s attempt to formalize universality. I think a “belief” is not the right language for an object of a system for two reasons. First, it conflates terms worth making distinct. When “beliefs” are used in Paul Christiano’s attempt to formalize universality, they don’t always refer to the same thing. Second, “belief” should be reserved for a different concept. The way humans think of “belief” is not how it’s used there.

So, I rewrote the formalization of universality to not include the word “belief”. The handful of terms presented below combine to replace the previous notion of “belief”. Presenting universality without the language of “beliefs”, all said, is certainly more cumbersome.

I hope this reframing will help improve the methodology for applying universality to HCH. I apply the general definitions below to universality and the Filter in section III.2.

2. General definitions for universality

Consider κ, a list of action-shaping intelligible information. κ={κj}, where κj are pieces of intelligible information. Pieces of intelligible information are facts, which are sentences (i.e. “The sky is blue”, “there are infinitely many twin primes”). Another term, X, is the output of a computation. X is a list of actions, X={xl} (i.e. Printed to my computer the sentence “the sky is blue”, wiped the memory of Alice’s computer next door). A view may be modeled One may view a computation A as, upon an input of the world γ, taking a set of actions X. A(γ)=X. κ is a part of γ; action-shaping intelligible information of a system is part of the input of the world which determines the actions the system will take.

More formally:

Now, what was previously known as “belief” is now usually denoted as an action-shaping list of intelligible information, κ={κj}Jj=j0. In item II (above), a formal constraint of causality requires some causal theory, and also importing some causal assumptions. I do not provide this here. A causal constraint is necessary in addition to the statement A(κ)=xl to preclude the inclusion of arbitrary information in κ which are not related to the actions xl and which, without this constraint, are permitted to be very numerous.

I expect that item III (above) is closer to a common rule of thumb than to a universal truth. Some theories of causality between κ and xl might afford that in special circumstances, κ is unique (N=1).

Then, I may present universality with a couple formulas:

Note that the first and third statements are directly transcribed from Paul Christiano’s attempt to formalize universality.

The re...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Motivations, Natural Selection, and Curriculum Engineering, published by Oliver Sourbut on December 16, 2021 on The AI Alignment Forum.

Epistemic status: I am professionally familiar with contemporary machine learning theory and practice, and have a long-standing amateur interest in natural history. But (despite being a long-term lurker on LW and AF) I feel very uncertain about the nature and status of goals and motivations! This post is an attempt, heavily inspired by Richard Ngo’s Shaping Safer Goals sequence, to lay out a few ways of thinking about goals and motivations in natural and artificial systems, to point to a class of possible prosaic development paths to AGI where general capabilities emerge from extensive open-ended training, and finally observe some desiderata and gaps in our understanding regarding alignment conditional on this situation.

This post was supported by the Stanford Existential Risks Initiative Machine Learning Alignment Scholars (MATS) program and I thank them for their support, Evan Hubinger for his mentorship during the project, and Richard Ngo and Oliver Zhang for helpful discussion and feedback on portions of the post.

Abstract/TL;DR

Recent capabilities progress, scaling and complexity observations, and (pre)historical precedent all suggest that open-ended training of baseline/foundation models is emerging as a plausible ‘very prosaic’ route to AGI. Multi-agent environments may include a particularly rich class of training regimes. Any generally-applicable capabilities such foundation models develop may by default come along with ‘core motivations’ which develop over the training process. We can not necessarily expect those motivations to generalise in aligned ways at deployment.

There are many free variables in the regimes described in the preceding claims, including end-to-endness, diversity and kinds of training curricula, extent and form of human participation, sizes and homogeneity of populations, architectures, means of information transmission, etc. We would gain from investment in exploring the space, the better to develop techniques for making the route safer, and the better to steer through safer sub-routes (or motivate opposition to this route entirely).

This post begins by elaborating and challenging the above framing. Then some suggestions are made for ways of breaking down the concept of motivations, followed by a presentation of some considerations and desiderata for how we might influence them in the above setting. Genetic natural selection and its products are used as a substantial anchoring point for predictions, while attempting to include a diversity of sources and examples from natural history, as well as scrutinising the (dis)analogy and departing from it where possible.

Connections are made to the concept of ‘corrigibility’ as discussed by various authors, and value receptivity and imprinting are offered as some particular points of potential inspiration from nature for alignment. The phylogenetic-ontogenetic distinction is explored as an analogy for mechanically different portions of development stories for artificial systems.

How to Read This Post

For those who have read Shaping Safer Goals, sections Explicit Pressure on Motivation Types, Implicit Pressure on Motivation Types, Modularity of Capability Accumulation, and Collectives should seem familiar, as they are each maybe 50-80% summary of Richard’s discussions there, with framing and discussion added. Early vs Late Motivations and Training Duration also contains some summary as do the other sections to varying degrees.

Section Intro and Why to Care is intended to be as relevant to that sequence as it is to the rest of this post, and may be a good place to start if you intend to read either or both, or if you want more context to understand Richard’s sequen...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Motivations, Natural Selection, and Curriculum Engineering, published by Oliver Sourbut on December 16, 2021 on The AI Alignment Forum.

Epistemic status: I am professionally familiar with contemporary machine learning theory and practice, and have a long-standing amateur interest in natural history. But (despite being a long-term lurker on LW and AF) I feel very uncertain about the nature and status of goals and motivations! This post is an attempt, heavily inspired by Richard Ngo’s Shaping Safer Goals sequence, to lay out a few ways of thinking about goals and motivations in natural and artificial systems, to point to a class of possible prosaic development paths to AGI where general capabilities emerge from extensive open-ended training, and finally observe some desiderata and gaps in our understanding regarding alignment conditional on this situation.

This post was supported by the Stanford Existential Risks Initiative Machine Learning Alignment Scholars (MATS) program and I thank them for their support, Evan Hubinger for his mentorship during the project, and Richard Ngo and Oliver Zhang for helpful discussion and feedback on portions of the post.

Abstract/TL;DR

Recent capabilities progress, scaling and complexity observations, and (pre)historical precedent all suggest that open-ended training of baseline/foundation models is emerging as a plausible ‘very prosaic’ route to AGI. Multi-agent environments may include a particularly rich class of training regimes. Any generally-applicable capabilities such foundation models develop may by default come along with ‘core motivations’ which develop over the training process. We can not necessarily expect those motivations to generalise in aligned ways at deployment.

There are many free variables in the regimes described in the preceding claims, including end-to-endness, diversity and kinds of training curricula, extent and form of human participation, sizes and homogeneity of populations, architectures, means of information transmission, etc. We would gain from investment in exploring the space, the better to develop techniques for making the route safer, and the better to steer through safer sub-routes (or motivate opposition to this route entirely).

This post begins by elaborating and challenging the above framing. Then some suggestions are made for ways of breaking down the concept of motivations, followed by a presentation of some considerations and desiderata for how we might influence them in the above setting. Genetic natural selection and its products are used as a substantial anchoring point for predictions, while attempting to include a diversity of sources and examples from natural history, as well as scrutinising the (dis)analogy and departing from it where possible.

Connections are made to the concept of ‘corrigibility’ as discussed by various authors, and value receptivity and imprinting are offered as some particular points of potential inspiration from nature for alignment. The phylogenetic-ontogenetic distinction is explored as an analogy for mechanically different portions of development stories for artificial systems.

How to Read This Post

For those who have read Shaping Safer Goals, sections Explicit Pressure on Motivation Types, Implicit Pressure on Motivation Types, Modularity of Capability Accumulation, and Collectives should seem familiar, as they are each maybe 50-80% summary of Richard’s discussions there, with framing and discussion added. Early vs Late Motivations and Training Duration also contains some summary as do the other sections to varying degrees.

Section Intro and Why to Care is intended to be as relevant to that sequence as it is to the rest of this post, and may be a good place to start if you intend to read either or both, or if you want more context to understand Richard’s sequen...