Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Theoretical Neuroscience For Alignment Theory, published by Cameron Berg on December 7, 2021 on LessWrong.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

Many additional thanks to Steve Byrnes and Adam Shimi for their helpful feedback on earlier drafts of this post.

TL;DR: Steve Byrnes has done really exciting work at the intersection of neuroscience and alignment theory. He argues that because we’re probably going to end up at some point with an AGI whose subparts at least superficially resemble those of the brain (a value function, a world model, etc.), it’s really important for alignment to proactively understand how the many ML-like algorithms in the brain actually do their thing. I build off of Steve’s framework in the second half of this post: first, I discuss why it would be worthwhile to understand the computations that underlie theory of mind + affective empathy. Second, I introduce the problem of self-referential misalignment, which is essentially the worry that initially-aligned ML systems with the capacity to model their own values could assign second-order values to these models that ultimately result in contradictory—and thus misaligned—behavioral policies.

(A simple example of this general phenomenon in humans: Jack hates reading fiction, but Jack wants to be the kind of guy who likes reading fiction, so he forces himself to read fiction.)

Introduction

In this post, my goal is to distill and expand upon some of Steve Byrnes’s thinking on AGI safety. For those unfamiliar with his work, Steve thinks about alignment largely through the lens of his own brand of “big-picture” theoretical neuroscience. Many of his formulations in this space are thus original and ever-evolving, which is all the more reason to attempt to consolidate his core ideas in one space. I’ll begin by summarizing Steve’s general perspectives on AGI safety and threat models. I’ll then turn to Steve’s various models of the brain and its neuromodulatory systems and how these conceptualizations relate to AGI safety. In the second half of this post, I’ll spend time exploring two novel directions for alignment theory that I think naturally emerge from Steve’s thinking.

Steve’s framework

Steve’s work in alignment theory

In order to build a coherent threat model (and before we start explicitly thinking about any brain-based algorithms), Steve reasons that we first need to operationalize some basic idea of what components we would expect to constitute an AGI. Steve asserts that four ingredients seem especially likely: a world model, a value function, a planner/actor, and a reward function calculator. As such, he imagines AGI to be fundamentally grounded in model-based RL.

So, in the simple example of an agent navigating a maze, the world model would be some learned map of that maze, the value function might assign values to every juncture (e.g., turning left here = +5, turning right here = -5), the planner/actor would transmute these values into a behavioral trajectory, and the reward function calculator would translate certain outcomes of that trajectory into rewards for the agent (e.g., +10 for successfully reaching the end of the maze). Note here that the first three ingredients of this generic AGI—its world model, value function, and planner/actor—are presumed to be learned, while the reward function calculator is considered to be hardcoded or otherwise fixed. This distinction (reward function = fixed; everything else = learned) will be critical for understanding Steve’s subsequent thinking in AGI safety and his motivations for studying neuroscience.

Threat models

Using these algorithmic ingredients, Steve recasts inner alignment to simply refer to cases where an AG...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Theoretical Neuroscience For Alignment Theory, published by Cameron Berg on December 7, 2021 on LessWrong.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

Many additional thanks to Steve Byrnes and Adam Shimi for their helpful feedback on earlier drafts of this post.

TL;DR: Steve Byrnes has done really exciting work at the intersection of neuroscience and alignment theory. He argues that because we’re probably going to end up at some point with an AGI whose subparts at least superficially resemble those of the brain (a value function, a world model, etc.), it’s really important for alignment to proactively understand how the many ML-like algorithms in the brain actually do their thing. I build off of Steve’s framework in the second half of this post: first, I discuss why it would be worthwhile to understand the computations that underlie theory of mind + affective empathy. Second, I introduce the problem of self-referential misalignment, which is essentially the worry that initially-aligned ML systems with the capacity to model their own values could assign second-order values to these models that ultimately result in contradictory—and thus misaligned—behavioral policies.

(A simple example of this general phenomenon in humans: Jack hates reading fiction, but Jack wants to be the kind of guy who likes reading fiction, so he forces himself to read fiction.)

Introduction

In this post, my goal is to distill and expand upon some of Steve Byrnes’s thinking on AGI safety. For those unfamiliar with his work, Steve thinks about alignment largely through the lens of his own brand of “big-picture” theoretical neuroscience. Many of his formulations in this space are thus original and ever-evolving, which is all the more reason to attempt to consolidate his core ideas in one space. I’ll begin by summarizing Steve’s general perspectives on AGI safety and threat models. I’ll then turn to Steve’s various models of the brain and its neuromodulatory systems and how these conceptualizations relate to AGI safety. In the second half of this post, I’ll spend time exploring two novel directions for alignment theory that I think naturally emerge from Steve’s thinking.

Steve’s framework

Steve’s work in alignment theory

In order to build a coherent threat model (and before we start explicitly thinking about any brain-based algorithms), Steve reasons that we first need to operationalize some basic idea of what components we would expect to constitute an AGI. Steve asserts that four ingredients seem especially likely: a world model, a value function, a planner/actor, and a reward function calculator. As such, he imagines AGI to be fundamentally grounded in model-based RL.

So, in the simple example of an agent navigating a maze, the world model would be some learned map of that maze, the value function might assign values to every juncture (e.g., turning left here = +5, turning right here = -5), the planner/actor would transmute these values into a behavioral trajectory, and the reward function calculator would translate certain outcomes of that trajectory into rewards for the agent (e.g., +10 for successfully reaching the end of the maze). Note here that the first three ingredients of this generic AGI—its world model, value function, and planner/actor—are presumed to be learned, while the reward function calculator is considered to be hardcoded or otherwise fixed. This distinction (reward function = fixed; everything else = learned) will be critical for understanding Steve’s subsequent thinking in AGI safety and his motivations for studying neuroscience.

Threat models

Using these algorithmic ingredients, Steve recasts inner alignment to simply refer to cases where an AG...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interviews on Improving the AI Safety Pipeline, published by Chris Leong on December 7, 2021 on LessWrong.

I conducted the following interviews to better understand how we might aim to improve the AI safety pipeline. It is very hard to summarise these interviews as everyone has different ideas of what needs to be done. However, I suppose there is some value in knowing this in and of itself.

A lot of people think that we ought to grow the field more, but this is exactly the kind of proposal that we should definitely not judge based on popularity. Given the number of people trying to break into the field, such a proposal was almost automatically going to be popular. I suppose a more useful takeaway is that a lot of people are currently seriously pursuing proposals to run projects in this space.

I guess another comment I can make is that the people who are most involved in the field seemed quite enthusiastic about the Cambridge AI Safety Fundamentals course as a way of growing the field.

If you think you’d have interesting thoughts to share, please feel free to PM me and we can set up a meeting.

Disclaimers: These interviews definitely aren’t a random sample. Some of the people were just people I wanted to talk to anyway. I was also reluctant to contact really senior people as I didn’t want to waste their time.

I tended towards being inclusive, so the mere fact that I talked to someone or included some of what they shared shouldn’t be taken as an endorsement of their views or of them being any kind of authority.

Evan Hubringer (MIRI):

Evan is now running a mentorship program with the assistance of SERI to mentor more people that he would be able to mentor himself:

Participants (14 people) were recruited from the AI Safety Fundamentals Course

This involves going through a set of shared readings for a few months, summarising and expanding upon a research artifact, then attempting an open research problem

SERI handled interviews + logistics

Evan had a 1/2 hour 1-on-1 weekly chat with each participant in the 1st stage & 1 hour weekly chat in the second stage where he cut the number of participants in half

Evan would like to see other researchers could copy this model to maximise their ability to mentor people:

Chris’ Note:

Perhaps Non-Linear could organise something similar under multipliers for existing talent - although I think Evan might try expanding this to other researchers himself

Suggested that there needs to be a new organisation to house people:

Why?:

None of the current places can scale rapidly

Research focus of MIRI is narrow

Structure:

Suggested a mix of remote and in-person

A comprehensive research agenda limits ability to scale

Challenges:

Hardest part is training. People need a model of AI safety and AI:

Easiest way to get it is to copy it from someone else

Can’t just read things as you need to be able to poke the model

Suggested that the EA movement has been the most successful way of getting people interested in AI Safety

Buck Shlegeris (Redwood Research):

Suggested that there need to be more problem sets:

These can be integrated into the middle of articles

Greatly impressed by the AGI Safety Fundamentals course - he liked the content and who it attracted

Running a machine learning bootcamp (details here):

Aim is to turn strong python programmers into ML engineers (handling the engineering parts of their research)

Surprisingly, full-stack web experience can be useful here such as if you’ve learned to handle large volumes of data

Expects to hire people in the following ratio: 1 researcher, 2 research engineers, 1 infrastructure

Someone involved in the Cambridge AGI Safety Fundamentals Course:

Fundamentals Course

Content covers:

Why AI risk might be particularly concerning. Introducing various research agendas.

Further course details here

Not optimised for g...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interviews on Improving the AI Safety Pipeline, published by Chris Leong on December 7, 2021 on LessWrong.

I conducted the following interviews to better understand how we might aim to improve the AI safety pipeline. It is very hard to summarise these interviews as everyone has different ideas of what needs to be done. However, I suppose there is some value in knowing this in and of itself.

A lot of people think that we ought to grow the field more, but this is exactly the kind of proposal that we should definitely not judge based on popularity. Given the number of people trying to break into the field, such a proposal was almost automatically going to be popular. I suppose a more useful takeaway is that a lot of people are currently seriously pursuing proposals to run projects in this space.

I guess another comment I can make is that the people who are most involved in the field seemed quite enthusiastic about the Cambridge AI Safety Fundamentals course as a way of growing the field.

If you think you’d have interesting thoughts to share, please feel free to PM me and we can set up a meeting.

Disclaimers: These interviews definitely aren’t a random sample. Some of the people were just people I wanted to talk to anyway. I was also reluctant to contact really senior people as I didn’t want to waste their time.

I tended towards being inclusive, so the mere fact that I talked to someone or included some of what they shared shouldn’t be taken as an endorsement of their views or of them being any kind of authority.

Evan Hubringer (MIRI):

Evan is now running a mentorship program with the assistance of SERI to mentor more people that he would be able to mentor himself:

Participants (14 people) were recruited from the AI Safety Fundamentals Course

This involves going through a set of shared readings for a few months, summarising and expanding upon a research artifact, then attempting an open research problem

SERI handled interviews + logistics

Evan had a 1/2 hour 1-on-1 weekly chat with each participant in the 1st stage & 1 hour weekly chat in the second stage where he cut the number of participants in half

Evan would like to see other researchers could copy this model to maximise their ability to mentor people:

Chris’ Note:

Perhaps Non-Linear could organise something similar under multipliers for existing talent - although I think Evan might try expanding this to other researchers himself

Suggested that there needs to be a new organisation to house people:

Why?:

None of the current places can scale rapidly

Research focus of MIRI is narrow

Structure:

Suggested a mix of remote and in-person

A comprehensive research agenda limits ability to scale

Challenges:

Hardest part is training. People need a model of AI safety and AI:

Easiest way to get it is to copy it from someone else

Can’t just read things as you need to be able to poke the model

Suggested that the EA movement has been the most successful way of getting people interested in AI Safety

Buck Shlegeris (Redwood Research):

Suggested that there need to be more problem sets:

These can be integrated into the middle of articles

Greatly impressed by the AGI Safety Fundamentals course - he liked the content and who it attracted

Running a machine learning bootcamp (details here):

Aim is to turn strong python programmers into ML engineers (handling the engineering parts of their research)

Surprisingly, full-stack web experience can be useful here such as if you’ve learned to handle large volumes of data

Expects to hire people in the following ratio: 1 researcher, 2 research engineers, 1 infrastructure

Someone involved in the Cambridge AGI Safety Fundamentals Course:

Fundamentals Course

Content covers:

Why AI risk might be particularly concerning. Introducing various research agendas.

Further course details here

Not optimised for g...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Let's buy out Cyc, for use in AGI interpretability systems?, published by Steven Byrnes on December 7, 2021 on LessWrong.

I hear that we're supposed to brainstorm ways to spend massive amounts of money to advance AGI safety. Well here's a brainstorm, sorry if it's stupid, I'm not an interpretability researcher:

We could pay Cycorp to open-source Cyc, so that researchers can incorporate it into future AGI interpretability systems.

What is Cyc? I'm not an expert. Most of what I know about Cyc comes from this podcast interview, and wikipedia, and LW wiki, and Eliezer Yudkowsky dismissing Cyc as a not on the path to AGI. I agree, by the way: I don't think that Cyc is on the path to AGI. (I'm not even sure if it's trying to be.) But it doesn't matter, that's not why I'm talking about it.

Anyway, Cyc is a language for expressing "knowledge" (e.g. water will fall out of an upside-down cup), and super-giant database of such "knowledge", hand-coded over the course of >1000 person-years and >35 wall-clock years, by a team of tireless employees whom I do not envy.

How expensive would this be: Beats me. I didn't ask. (Wild guess: one-time cost in the eight figures, based on Cycorp annual revenue of ~$5M according to the first hit on google (which I didn't double-check).)

Why might open-sourcing Cyc be helpful for AGI interpretability efforts? Well maybe it won't. But FWIW, here's what I was thinking.

When I imagine what a future interpretability system will look like, in general, I imagine an interface.

The human-legible side of the interface consists of, maybe, human-understandable natural-language phrases, or human-understandable pictures, or human-understandable equations, or whatever.

The trained-model side of the interface consists of stuff that's happening in a big complicated unlabeled model built by ML.

Anyway, my surmise is that Cyc might be a good fit for the former (human-legible) side of this interface.

Specifically, the Cyc project has this massive structured knowledge system, and everything in that system is human-legible. It was, after all, built by humans!

You might say: natural language is human-legible too. Why not use that?

Well for one thing, natural-language descriptions can be ambiguous. For example, dictionary words may have dozens of definitions. In Cyc, such a word would be disambiguated: it would be dozens of different tokens for the dozens of different definitions and nuances. (.If I understand correctly. It came up in the interview.)

For another thing, there's a matching-across-the-interface challenge: we want to make sure that the right human-legible bits are matched to the corresponding trained-model bits (at least, insofar as such a thing is possible). Here, the massive Cyc knowledge database (i.e. millions of common-sense facts about the world like "air can pass through a screen door" or whatever else those poor employees had to input over the decades) would presumably come in handy. After building a link between things in the ML system and Cyc, we can check (somehow) that the ML system endorses all the "knowledge" in Cyc as being correct. If that test passes, then that's a good sign we're matching the two things up properly; if it fails in some area, we would want to go back and figure out what's going on there.

I don't want to make too strong a statement though. At the end of the day, it seems to me that a natural-language interface to a future AGI interpretability system has both strengths and weaknesses compared to a Cyc interface. (And there are other possibilities too.) Hey, maybe we should have both! When it comes to interpretability, I say more dakka :-P

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Let's buy out Cyc, for use in AGI interpretability systems?, published by Steven Byrnes on December 7, 2021 on LessWrong.

I hear that we're supposed to brainstorm ways to spend massive amounts of money to advance AGI safety. Well here's a brainstorm, sorry if it's stupid, I'm not an interpretability researcher:

We could pay Cycorp to open-source Cyc, so that researchers can incorporate it into future AGI interpretability systems.

What is Cyc? I'm not an expert. Most of what I know about Cyc comes from this podcast interview, and wikipedia, and LW wiki, and Eliezer Yudkowsky dismissing Cyc as a not on the path to AGI. I agree, by the way: I don't think that Cyc is on the path to AGI. (I'm not even sure if it's trying to be.) But it doesn't matter, that's not why I'm talking about it.

Anyway, Cyc is a language for expressing "knowledge" (e.g. water will fall out of an upside-down cup), and super-giant database of such "knowledge", hand-coded over the course of >1000 person-years and >35 wall-clock years, by a team of tireless employees whom I do not envy.

How expensive would this be: Beats me. I didn't ask. (Wild guess: one-time cost in the eight figures, based on Cycorp annual revenue of ~$5M according to the first hit on google (which I didn't double-check).)

Why might open-sourcing Cyc be helpful for AGI interpretability efforts? Well maybe it won't. But FWIW, here's what I was thinking.

When I imagine what a future interpretability system will look like, in general, I imagine an interface.

The human-legible side of the interface consists of, maybe, human-understandable natural-language phrases, or human-understandable pictures, or human-understandable equations, or whatever.

The trained-model side of the interface consists of stuff that's happening in a big complicated unlabeled model built by ML.

Anyway, my surmise is that Cyc might be a good fit for the former (human-legible) side of this interface.

Specifically, the Cyc project has this massive structured knowledge system, and everything in that system is human-legible. It was, after all, built by humans!

You might say: natural language is human-legible too. Why not use that?

Well for one thing, natural-language descriptions can be ambiguous. For example, dictionary words may have dozens of definitions. In Cyc, such a word would be disambiguated: it would be dozens of different tokens for the dozens of different definitions and nuances. (.If I understand correctly. It came up in the interview.)

For another thing, there's a matching-across-the-interface challenge: we want to make sure that the right human-legible bits are matched to the corresponding trained-model bits (at least, insofar as such a thing is possible). Here, the massive Cyc knowledge database (i.e. millions of common-sense facts about the world like "air can pass through a screen door" or whatever else those poor employees had to input over the decades) would presumably come in handy. After building a link between things in the ML system and Cyc, we can check (somehow) that the ML system endorses all the "knowledge" in Cyc as being correct. If that test passes, then that's a good sign we're matching the two things up properly; if it fails in some area, we would want to go back and figure out what's going on there.

I don't want to make too strong a statement though. At the end of the day, it seems to me that a natural-language interface to a future AGI interpretability system has both strengths and weaknesses compared to a Cyc interface. (And there are other possibilities too.) Hey, maybe we should have both! When it comes to interpretability, I say more dakka :-P

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Counting Lightning, published by Jsevillamol on December 7, 2021 on LessWrong.

When Samir returned to the base JiuJang waited by the entrance. There was no point in denying he had disobeyed JiuJang's warning, so he was upfront about it.

"I went outside," Samir admitted. "It was raining heavily. The wind blew so strong I barely could hear anything. And every second or so, lightning. I kept waiting for thunder to come, but there was none."

JiuJang sat down, and sunk his face between his arms.

"It is not lightning", JiuJang said. "You cannot count lightning".

"What do you mean?" Asked Samir.

JiuJang didn't respond.

Through the tinted glass, Samir could just barely notice the pulses of light. He tried counting them.

One. Another. And a third.

Then he saw it. One, two. One, two. A pattern, instead of randomness. Not lightning then.

"What does it mean?" Samir asked once more.

"It means you have doomed you and your people," JiuJang answered, somber.

One, two. One, two. The lights became brighter with every flash. Getting closer.

Samir panicked. He looked out of the window, trying to distinguish movement in the shadows. Then he ran inside the base, to alert the rest of the refugees.

JiuJang sat and waited. He knew it was pointless to do otherwise.

Seconds after, the windows broke. The drones, as always, ignored him, and lurched inside.

He heard screams. Shots. Then silence.

The drones flew out, orderly.

When the pulses grew dimmer, JiuJang descended into the basement. The door was out of its frame.

On the other side, Samir lay dead. His body was covering a smaller one - his daughter, Alotta. She was alive, but had been shot in her lungs.

Xianxi, JiuJang's wife, was trying to get Samir's corpse off her daughter. JiuJang helped her, and she rushed to cover her wounds. Xianxi was no doctor; Alotta would not live long.

A dozen other corpses were scattered throughout the room. Their faces were frozen in terror.

JiuJang sighed, shed a tear and grabbed a shovel. There were graves to be dug.

Stella zipped through the factory on her rollerblades.

She moved carefully and she moved fast. People were counting on her. Hopefully the owners were not.

She took a left and then a sharp turn right. She had memorized the schematics, and was anticipating every movement she would have to make until reaching the neural processor assembly line.

What she didn't expect was a cleaning drone in the middle of the hall.

Stella crashed against it with an unceremonious thud, and fell to the floor. She grunted.

The drone turned around, and her heart stopped beating. Then the drone turned back, and kept sweeping the floor.

Right, of course. She was wearing her seals. Any computer would recognise her as a stray cat. At least the ones using the latest patch of openCV software.

Here, she was invisible.

Stella got up and continued like an arrow.

The door to the assembly line was locked. Voice activated; not a problem for a witch of her caliber.

She took her phone out and typed the name of a script. She spoke into it, and out came her voice, disguised in the pitch of one of the factory executives.

A voice sample for fine tuning hadn't been hard to come by. He had come out in television a few days back, testifying about the slaughterbot genocide in western China.

The door opened with a click. Abracadabra. Stella let herself in, and marveled at the assembly line.

The factory was fully automated. Cameras oversaw the operation, and in theory so did the engineers sitting behind the cameras. In practice, they rarely bothered to check; there was rarely a need to.

Stella was after the neural architecture the bots were using for face recognition. With that, her coven would be able to make seals of protection for the refugees.

Not all of them. If the kill count got too low the military would realise, and quickly find the i...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Counting Lightning, published by Jsevillamol on December 7, 2021 on LessWrong.

When Samir returned to the base JiuJang waited by the entrance. There was no point in denying he had disobeyed JiuJang's warning, so he was upfront about it.

"I went outside," Samir admitted. "It was raining heavily. The wind blew so strong I barely could hear anything. And every second or so, lightning. I kept waiting for thunder to come, but there was none."

JiuJang sat down, and sunk his face between his arms.

"It is not lightning", JiuJang said. "You cannot count lightning".

"What do you mean?" Asked Samir.

JiuJang didn't respond.

Through the tinted glass, Samir could just barely notice the pulses of light. He tried counting them.

One. Another. And a third.

Then he saw it. One, two. One, two. A pattern, instead of randomness. Not lightning then.

"What does it mean?" Samir asked once more.

"It means you have doomed you and your people," JiuJang answered, somber.

One, two. One, two. The lights became brighter with every flash. Getting closer.

Samir panicked. He looked out of the window, trying to distinguish movement in the shadows. Then he ran inside the base, to alert the rest of the refugees.

JiuJang sat and waited. He knew it was pointless to do otherwise.

Seconds after, the windows broke. The drones, as always, ignored him, and lurched inside.

He heard screams. Shots. Then silence.

The drones flew out, orderly.

When the pulses grew dimmer, JiuJang descended into the basement. The door was out of its frame.

On the other side, Samir lay dead. His body was covering a smaller one - his daughter, Alotta. She was alive, but had been shot in her lungs.

Xianxi, JiuJang's wife, was trying to get Samir's corpse off her daughter. JiuJang helped her, and she rushed to cover her wounds. Xianxi was no doctor; Alotta would not live long.

A dozen other corpses were scattered throughout the room. Their faces were frozen in terror.

JiuJang sighed, shed a tear and grabbed a shovel. There were graves to be dug.

Stella zipped through the factory on her rollerblades.

She moved carefully and she moved fast. People were counting on her. Hopefully the owners were not.

She took a left and then a sharp turn right. She had memorized the schematics, and was anticipating every movement she would have to make until reaching the neural processor assembly line.

What she didn't expect was a cleaning drone in the middle of the hall.

Stella crashed against it with an unceremonious thud, and fell to the floor. She grunted.

The drone turned around, and her heart stopped beating. Then the drone turned back, and kept sweeping the floor.

Right, of course. She was wearing her seals. Any computer would recognise her as a stray cat. At least the ones using the latest patch of openCV software.

Here, she was invisible.

Stella got up and continued like an arrow.

The door to the assembly line was locked. Voice activated; not a problem for a witch of her caliber.

She took her phone out and typed the name of a script. She spoke into it, and out came her voice, disguised in the pitch of one of the factory executives.

A voice sample for fine tuning hadn't been hard to come by. He had come out in television a few days back, testifying about the slaughterbot genocide in western China.

The door opened with a click. Abracadabra. Stella let herself in, and marveled at the assembly line.

The factory was fully automated. Cameras oversaw the operation, and in theory so did the engineers sitting behind the cameras. In practice, they rarely bothered to check; there was rarely a need to.

Stella was after the neural architecture the bots were using for face recognition. With that, her coven would be able to make seals of protection for the refugees.

Not all of them. If the kill count got too low the military would realise, and quickly find the i...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some thoughts on why adversarial training might be useful, published by Beth Barnes on December 8, 2021 on The AI Alignment Forum.

What are the reasons we might want to do adversarial training?

Here’s a rough taxonomy.

Your model has learnt a stupid heuristic instead of the thing you want It has overfit to whatever particular dataset you trained on. It is best described as ‘trying’ to pursue this particular heuristic, and it appears to be doing well because of some arbitrary correlation in your dataset. For example, it is succeeding at image classification by attending to high-frequency noise in the image. Doing adversarial training is useful because it finds examples where this heuristic is uncorrelated with what you want. This allows you to spot the problem and to improve the model’s alignment. It doesn’t matter if the examples are particularly consequential failures; it just matters that they help the model learn the correct pattern. The adversarial examples you find are a ‘surprise’ to the model.

Your model is deceptively alignedYour model is fairly well-described as trying to pursue some goal that is unaligned with you. It performs well on your training set because this is instrumentally useful. It plans to defect in particular situations where this is advantageous. Doing adversarial training is useful if you can use it to fool the model into thinking it’s in some situation where it should defect. This allows you to spot that it is deceptive, and maybe train this deception away. It doesn’t matter if the example involves the model causing a particularly bad thing to happen, it just matters that you are able to detect the deception. The adversarial examples you find are not a ‘surprise’ to the model -- it was deliberately planning to defect in some situations.

Your model is generally aligned, but an adversary (or some extremely unlikely input) can trigger temporary misaligned behaviour or even permanent misalignment. For example, an adversarial AI can present your model with some sequence of inputs that cause it to start ‘trying’ to help this AI instead of you. Your model ‘wants’ to avoid this and doesn’t know in advance which situations will induce this. Adversarial training is useful to find these situations and fix these vulnerabilities. It is important to find the most consequential failures - you care less about situations where the model is briefly unaligned but doesn’t do significant harm.

Your model is fully aligned under all circumstances, but sometimes does bad things because of incompetence. For example, sometimes it crashes a car because of a failure to recognise another car coming towards it. Adversarial training is useful to find cases where this happens. It is important to find the most consequential failures - you care less about situations where the model is briefly incompetent but doesn’t do significant harm. The adversarial examples you find are a ‘surprise’ to the model.

Aside: defining alignment

One confusing edge case for distinguishing alignment failure from capabilities failure is if the model ‘knows’ some concept, but doesn’t ‘know’ that that was the concept you wanted.

E.g. suppose the model classifies someone getting shot with a gun as not an injury. There are 3 possibilities:

The model does not have a reasonable concept of injury that includes getting hit by a bullet, or doesn’t know that getting shot at would cause you to get hit by a bullet

The model has this concept of injury, and understands that it will occur, but ‘thinks’ you only care about injuries involving swords or knives because almost all injuries in your training data involved those

The model understands what you want, but is pursuing some other goal

I think the first possibility is definitely a capabilities failure, and the last is definitely an alignment failure, but I’m...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some thoughts on why adversarial training might be useful, published by Beth Barnes on December 8, 2021 on The AI Alignment Forum.

What are the reasons we might want to do adversarial training?

Here’s a rough taxonomy.

Your model has learnt a stupid heuristic instead of the thing you want It has overfit to whatever particular dataset you trained on. It is best described as ‘trying’ to pursue this particular heuristic, and it appears to be doing well because of some arbitrary correlation in your dataset. For example, it is succeeding at image classification by attending to high-frequency noise in the image. Doing adversarial training is useful because it finds examples where this heuristic is uncorrelated with what you want. This allows you to spot the problem and to improve the model’s alignment. It doesn’t matter if the examples are particularly consequential failures; it just matters that they help the model learn the correct pattern. The adversarial examples you find are a ‘surprise’ to the model.

Your model is deceptively alignedYour model is fairly well-described as trying to pursue some goal that is unaligned with you. It performs well on your training set because this is instrumentally useful. It plans to defect in particular situations where this is advantageous. Doing adversarial training is useful if you can use it to fool the model into thinking it’s in some situation where it should defect. This allows you to spot that it is deceptive, and maybe train this deception away. It doesn’t matter if the example involves the model causing a particularly bad thing to happen, it just matters that you are able to detect the deception. The adversarial examples you find are not a ‘surprise’ to the model -- it was deliberately planning to defect in some situations.

Your model is generally aligned, but an adversary (or some extremely unlikely input) can trigger temporary misaligned behaviour or even permanent misalignment. For example, an adversarial AI can present your model with some sequence of inputs that cause it to start ‘trying’ to help this AI instead of you. Your model ‘wants’ to avoid this and doesn’t know in advance which situations will induce this. Adversarial training is useful to find these situations and fix these vulnerabilities. It is important to find the most consequential failures - you care less about situations where the model is briefly unaligned but doesn’t do significant harm.

Your model is fully aligned under all circumstances, but sometimes does bad things because of incompetence. For example, sometimes it crashes a car because of a failure to recognise another car coming towards it. Adversarial training is useful to find cases where this happens. It is important to find the most consequential failures - you care less about situations where the model is briefly incompetent but doesn’t do significant harm. The adversarial examples you find are a ‘surprise’ to the model.

Aside: defining alignment

One confusing edge case for distinguishing alignment failure from capabilities failure is if the model ‘knows’ some concept, but doesn’t ‘know’ that that was the concept you wanted.

E.g. suppose the model classifies someone getting shot with a gun as not an injury. There are 3 possibilities:

The model does not have a reasonable concept of injury that includes getting hit by a bullet, or doesn’t know that getting shot at would cause you to get hit by a bullet

The model has this concept of injury, and understands that it will occur, but ‘thinks’ you only care about injuries involving swords or knives because almost all injuries in your training data involved those

The model understands what you want, but is pursuing some other goal

I think the first possibility is definitely a capabilities failure, and the last is definitely an alignment failure, but I’m...