Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Eli's review of "Is power-seeking AI an existential risk?", published by elifland on September 30, 2022 on The AI Alignment Forum.

See also Joe Carlsmith's report "Is power-seeking AI an existential risk" and previous reviews. I'll excerpt 2 portions of my review below.

Thinking about reaching goal states rather than avoiding catastrophe

I have a serious complaint around the framing of how the risk is decomposed, which may systematically bias Joe, reviewers and readers toward lower estimates of existential risk than is warranted. It’s similar to Nate Soares’ concern regarding the risk being decomposed conjunctively when it might be more natural to think of it disjunctively, but I’ll express it in my own words.

In one sentence, my concern is that the framing of the report and decomposition is more like “avoid existential catastrophe” than “achieve a state where existential catastrophe is extremely unlikely and we are fulfilling humanity’s potential”, and this will bias readers toward lower estimates.

When playing a strategy board game, it’s common to think in terms of “win conditions”: states of the game that you could get to where you’d feel pretty confident about your chances to win, even if they’re intermediate. This is often something like a set of cards that puts you in a very strong position to accumulate a lot of points. I claim that we should often be thinking about AI futures more like “win conditions” in board games and less like avoiding a negative outcome.

What are the win conditions we should be thinking about? The ultimate goal state is something like some number of aligned APS-AIs filling the universe with value and bringing the chance to essentially 0 that misaligned APS-AIs will be created and ruin the equilibrium. When reasoning about the probability that we will avoid existential risk from APS-AI, we should ultimately be chaining to this goal state in some sense. Similar to in board games, it might make sense to aim for intermediate stable-ish states which we feel pretty good about and think about the probability we can reach those: a possible example might be coordinating to get the major players to slow down on AI scaling, utilizing aligned non-APS-AIs to police potential PS-misaligned APS-AIs from popping up, and doing tons of alignment research via whole brain emulation and/or non-APS AIs (thanks to Joe for verbal discussion on this point).

But either way we should often be thinking about win conditions and goal states, rather than avoiding a negative outcome.

In contrast to the goal state style of thinking, Joe’s decomposition frames the task at hand as avoiding an undesirable state of existential catastrophe. This should technically be fine, but I think biases risk estimates downward:

When thinking through the probability of the premises leading up to an existential catastrophe, I think it’s very hard to intuitively take the number of relevant actors consideration into account properly. The decomposition seems to prompt the reader toward reasoning about how likely it is that a specific APS-AI system or set of systems will disempower humanity than how likely it is that we will get to a state where no actors will deploy APS-AI systems that disempower humanity.

As Nate Soares argued, people are often reluctant to give very high probabilities to premises as it feels overconfident. But (1) means that very high probabilities may be warranted if it’s hard to achieve a state where it’s very hard for new actors to develop a PS-misaligned APS-AI! Very high probabilities might be warranted, but are hard to reason about and feel overconfident.

I think there are some advantages to the “avoid existential catastrophe” framing, but it’s likely less useful on the whole and at a minimum should be paired and reconciled with probability estimates given the goal s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: (Structural) Stability of Coupled Optimizers, published by Paul Bricman on September 30, 2022 on The AI Alignment Forum.

This post is part of my hypothesis subspace sequence, a living collection of proposals I'm exploring at Refine. Preceded by an interlude.

Thanks Adam Shimi, @artaxerxes, Herbert Jaeger, Tamsin Leake, and Ze Shen for discussion informing this post.

Intro

In the previous post, I explored a grammar of alignment proposals based on (what I later learned could be referred to as) coupled dynamical systems. To help you get a sense of this view, one family of proposals tackles the alignment problem by having human researchers optimize a trainer2 (e.g. oversight leagues) which optimizes a trainer (e.g. SGD across model space employing empirical risk as heuristic) which optimizes a model (e.g. LLM) which optimizes a world (e.g. text). Other families have different numbers of optimization levels (e.g. True Name partially obviates the need for trainer2), integrate feedback loops (e.g. human amplification), or extend the optimization stack in some other way. In this, the focus moves from what to point an AI at, to how to move towards the associated trainer state (e.g. using heuristics across trainer space and above, similar to empirical risk as heuristic across model space). A shift from targets to processes for reaching said targets.

From this level of abstraction, one can identify issues which systematically plague entire families of proposals. For instance, there's still a bounded human at the top guiding downstream optimization (e.g. on trainer2). In the current post, I aim to explore how two notions from dynamical systems apply at different levels of optimization, and how those might help alleviate systematic issues. The two notions are:

Stability. A region of the state space of a dynamical system is deemed stable if slight perturbations to objects in that region don't result in the object permanently leaving the region. For instance, a tiny asteroid which crashes into Earth won't cause Earth to leave its orbit around the Sun, because Earth's orbit is stable. In contrast, the smallest gust of wind blowing on a pencil balanced on its tip will cause it to leave that region of state space behind, as it's not stable. Note: While stability is mathematically defined only for such regions of state space, I'll stretch the term "stable" for the purpose of this post to also describe the dynamical system which in turn contains reachable stable solutions. This reading of stability can also be described as "the region of plausible initial conditions is contained within the basin of attraction of a single attractor." I purposefully avoided the related reading of "sensitive to initial conditions" due to its chaotic connotations.

Structural Stability. While the previous property touches mostly on perturbations of objects inhabiting a state space, structural stability touches on perturbations of the dynamical system itself. For instance, the map "implemented" by the laws of physics across world states would be deemed structurally stable if slight changes in the values of fundamental physical constants wouldn't radically change the states towards which the world converges. You might still get planets orbiting around stars making up galaxies. In contrast, if the laws of physics across world states would turn out not to be structurally stable, the slightest change to the fundamental physical constants would yield radically unfamiliar universes.

Instances

1. Model

Let's explore the notions of stability and structural stability in the context of different levels of the optimizer stack. First, if a model is stable (according to the stretched definition), then the particular state the world is initially in doesn't have much of an influence on the state of the world the world will ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We all teach: here's how to do it better, published by Michael Noetel on September 30, 2022 on The Effective Altruism Forum.

Epistemic status: There’s consistent meta-analytic evidence for the interventions and models presented here. Still, that research is only one piece of your evidence-based decision-making. I’ve been directive for brevity’s sake. Use your judgement, adapt to the context, and let me know where you disagree in the comments.

Summary: we need to learn to teach better

Education is the #2 focus area for the Centre for Effective Altruism. Many of us educate, from field building to fundraisers, from coffees to conferences. As a result, I think we can do better by applying the meta-analyses on what works from educational psychology. We can use models and strategies that have been shown to work for hundreds of thousands of students, rather than feeling like community building is a whole new paradigm.

I think the EA community can make better decisions, and track their progress more effectively, with an evidence-based theory of change for community building.

One useful and robustly supported model, self-determination theory, suggests we can be more effective if we help community members:

Feel more competent by helping them developing valuable skills using evidence-based teaching strategies (rather than focusing so much on knowledge, especially via passive learning)

Feel more autonomy by making learning more meaningful and aligning motivation with their personal values and goals (rather than using quite so much guilt and fear)

Feel more connected by empathising and accepting their experiences, promoting collaboration, and providing relatable role models.

Some of these ideas seem obvious, but the community is still affected by burnout, imposter syndrome, and perceived rejection. By better supporting psychological needs, we can mitigate the risks of people bouncing off or burning out.

If we want to know how well our community building is working, I think measuring these needs will give us an upstream predictor of highly-engaged EAs

Of these psychological needs, I think community-building efforts in EA struggle mostly in promoting feelings of ‘competence’, and neglect the long list of evidence-based strategies for building skills.

Most efforts are currently reading, listening to talks, and engaging in discussions. There are far more effective methods of helping people learn valuable skills and aptitudes.

Primarily, we should give people more time doing hands-on practice with the most important skills in an environment where they get constructive feedback. We should also learn to better use multimedia.

I appreciate and admire all those who do community building. If I identify any groups that could improve, it’s because I think they have the skills and track record to do an incredible amount of good. I want them to thrive, and I hope they see any feedback here as constructive.

How to teach effectively, in five steps

For those of you who don’t care about the causal model, or the evidence, the following summary will explain what good teaching looks like. I think we could frame more community building events using these steps. Doing so will build capacity in the community, and build members’ motivation and engagement. This obviously isn’t everything we should be doing to build a community—social events, 1 on 1s, conferences, and reading groups still have their place. But, if we’re trying to build a capable and motivated group, we should try the following more often.

Generally, work backward from some new skills you’re going to help people learn. These are your learning objectives. Knowledge is obviously good, but is most valuable and motivating when connected to important skills.

For example, don’t say “learn about AI” but instead “make well-calibrated forecasts abou...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prioritizing Parental Sleep, published by jefftk on September 30, 2022 on LessWrong.

Overall I've really enjoyed being a parent, but poor sleep has been the hardest part. At times we've both been so tired that we weren't able to think clearly about how to fix the problem, which can be very tricky. We've figured this out more over time, however, and sleep has improved with each successive kid (n=3). Here's the main things that have worked for us, all in one place.

Note that kids and parents vary a lot: our three kids are different from each other, and your kids likely even more so. I'm hoping that many things on this list will be useful to many people, but it wouldn't be surprising for several of them to be a poor fit for any individual family. I've ordered them roughly down from the ones that I think are most likely to work for anyone who tries them. Perhaps unsurprisingly this is also roughly in the order of youngest to oldest; my impression is kids diverge more over time as their personalities come out.

Sleeping in multiple rooms. If the baby waking up means only interrupting one parents sleep that's about half as much sleep deprivation.

Blackout curtains. Small children typically wake up with the sun, which means you're up to. If you can keep their room solidly dark in the morning by blocking out the sun you can shift their schedule to whenever is most convenient for you, typically a later waking.

Using a Snoo automated bassinet. It gently rocks the baby and shushes them back to sleep when they wake. We used this with our youngest for the first six months. It worked very well, and automated almost all of the helping babies fall back asleep that I needed to do with the older two. We weaned her off it much more gradually than they recommended, with first weaning mode (stops rocking after the baby is asleep) and then running it without rocking.

Sleep training. At some point they no longer need to feed as often, but either don't know how to fall back asleep on their own at the end of a sleep cycle or would enjoy getting a cuddle before going back to sleep. I wrote about our experience with our oldest, and how her naps got immediately better. With our youngest, at 15m, we've recently dropped the last night feed, and are in the process of teaching her that she needs to go back to sleep on her own the whole night.

Letting kids know when they can get up. When kids are a bit older, they'll often wake up near the end of the night and not be sure whether it's morning. An "OK to wake" light (or a janky incorrect clock) can help them figure out whether to go back to sleep or get up.

Child proofing bedrooms. When they're old enough to no longer be in a crib, it's great if their bedroom is a place that they can play unsupervised in the morning without waking you up.

Teaching them not to wake you. When they're old enough to play in the rest of the house unsupervised but still sometimes play too loudly or fight, teaching them that in the morning this isn't ok. If they wake up before us they have to resolve conflicts independently and keep their volume down; if they don't, both of go back to not being able to leave their rooms until we're up. I think the last time we needed to enforce this they were maybe 4y and 6y.

Other things that can be helpful:

If things are really not going well, and you're both way too tired to figure things out, pick one of you to get caught up on sleep while the other covers. The one that's caught up can then figure out what needs changing, and make it up the unevenness later.

Some people have had good success with cosleeping, but while we started with the two older kids sleeping in an annex cosleeper we didn't do this with the youngest; the Snoo worked better for us. We will sometimes cosleep when traveling or in other unusual situations.

Tracking wake wi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bruce Wayne and the Cost of Inaction, published by TurnTrout on September 30, 2022 on LessWrong.

My brother (DivineMango) and I (TurnTrout) wrote this first arc of a Batman fanfiction in the summer of 2020.

1: Don't Look Away

“The man’s starving. That doesn’t change if you look away, Bruce.”

Hand wrapped firmly around his son’s shoulder, Thomas Wayne directed his son’s flinching gaze towards a haggard face. The man looked crumpled, his beard long and unkempt, his eyes bagged and bloodshot. A tattered cardboard sign hung heavily from his neck:

Bruce was scared. At eight, he’d seen homeless people before, but he’d never connected the dots: this was a person, just out on the street.

Thomas walked towards the man, his designer leather shoes tapping out a confident pace against the pavement. In a smooth motion, he pulled out his billfold and kneeled to speak.

The man flinched. He seemed surprised anyone had even noticed he was there, and his eyes widened as he recognized the magnate of Wayne Enterprises. Ignoring the looks shot his way by the affluent passersby, Thomas spoke with the man. After about a minute, he pressed two crisp large bills into the man’s dirty hands; between the bills was the business card of one of Gotham’s better psychiatrists.

As the father and son resumed their stroll, Bruce looked up at his dad, his eyes still wide.

“What happened to him? Did he used to be a kid like me?”

Thomas considered his response as they walked down the boulevard and past the gleaming storefronts. To their right stretched a verdant park dotted by dark wooden benches, some of which encircled a duck pond. Together, they sat.

“His name is Oscar, Bruce. He’s seen terrible things. Sometimes, people, good people, can get hurt in ways that don’t heal. It’s not like skinning your knee—it’s like falling down a dark well, and no one comes to help you climb out.”

“That’s awful.”

“Yes, it is. It is awful.”

“I don’t want to. I shouldn’t get any more toys until we help these people.”

Thomas blinked. The musical conversation of a happy couple drifted by. The sun cast long and deep shadows across the park.

“I’m proud of you, Bruce. Never forget that feeling. But... that’s not right. If you only let yourself be happy when everyone else in the world is, you’ll never smile again. The world is big, and no man or woman can fix everything by themselves.”

“Why not?”, Bruce asked earnestly, legs swinging lazily under the bench, feet grazing the tips of the grass.

“Why not?”, Thomas echoed softly. “Problems are hard. To land on the moon, we had to understand mathematics, and physics, and ten thousand details besides. To stop viruses, we had to understand our own bodies so well that we could teach them to attack the virus before it could hurt us. If you want to change the world, you have to understand it first.

“Even though we employ many skilled people, I can’t just cut a check for ‘fixing everything.’ If people get hurt and never heal, we need to understand why. Then, we need to change the system in just the right way, so that the problem goes away and never comes back. That’s hard work, and there are many problems.”

Bruce nodded. The sun winked over the skyline, and the sky waxed purple. Darkness would soon be upon them.

“Some people use this as an excuse. Since they can’t solve everything, they won’t try to solve anything.

“It’s easy to let details like Oscar fade away. You might notice a broken flower pot the first time you walk by, but by the third pass, your mind forgets to notice. But Oscar isn’t a flower pot. He was someone’s son. He had dreams, and heroes, and toys.”

The silver and black skyline glimmered against the velvet curtain sky. Silence overcame them for a moment, like the pause before a question's answer. Thomas' gaze meandered onto the nearest tower, which housed the main branch o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How To Observe Abstract Objects, published by LoganStrohl on September 30, 2022 on LessWrong.

Opening Thoughts

What is this thing and what is the point of it?

I’m trying to build a branch of rationality that is about looking at ideas and problems “for real”, thinking about them “for real”, “as a whole person”, and “without all the bullshit in the way”. This is a mini workshop in that vein.

The exercises here are about original seeing. They’re meant to help you stretch and strengthen a couple kinds of perceptual muscles. Most of them are not much like “here is how to do the thing”; they’re more like “here’s some stuff that might conceivably lead to you independently figuring out what the thing is and how to do it”. So be ready to experiment. Be ready to modify my instructions according to your whims.

This endeavor will happen in three phases.

Phase One takes about twenty minutes to complete, and stands alone pretty well. In it, you will directly observe a concrete object (something you can hold in your hands, like a carrot or a teacup). Phase Two takes about ten minutes, and leads you to summon an abstract object (something you can't hold in your hands, like scout mindset, the future, or whatever happens when you talk to your mother). Phase Three takes another ten minutes, and should be completed right after Phase Two. In Phase Three, you will directly observe an abstract object.

If you want to make a mini-workshop of this, block off an hour, and take breaks.

If all goes well, you'll leave with a greater ability to think about things originally. You'll be better at observing absolutely anything in ways that sometimes reveal new information you could not have uncovered through habit, convention, or rehearsal of your preconceptions.

Epistemic Status, History Of the Exercises, and Responsible Use

(Feel free to skip this part and go straight to Phase One, if the heading doesn’t interest you.)

I think the stuff in Phase One is moderately solid.

I first designed and ran something like it as a CFAR unit around 2018. I’ve been using it and tinkering with it since, in both pedagogical and personal contexts (n≈75 people), and it’s come to be an important part of how I guide people toward a more direct approach to thinking and solving problems. It’s only a first step into direct observation that’s probably not all that useful on its own, but I bet it’s a pretty good first step for most people.

However, a few people seem to have an overall cognitive strategy that crucially depends on not looking at things too closely (or something like that), and this is actively bad for some of them. If you try this for a minute and hate it, especially in an “I feel like I’m going crazy” kind of way, I do not recommend continuing. Go touch some grass instead. I’ve never seen this cause damage in just a few minutes (or at all, as far as I can tell), but I do think there’s a danger of dismantling somebody’s central coping mechanism if they push past their own red flags about it over and over again, or for a whole hour at once.

The stuff in Phases Two and Three is way more experimental. In fact, I’ve yet to run the full unit the same way twice, in the seven-ish times I’ve run it. I’ve tried these exercises in roughly this form with several people one-on one, with several small groups, and with one larger group, and I’m left with a sense that “this is roughly the right direction, but more work is needed”.

On the safety of Phases Two and Three: I haven’t seen any concerning-to-me reactions from people who have tried Phase Two or Phase Three, but I also have less data (n≈20). However, the exercises after Phase One are lighter touch—more of you feeling your own way around however you want, less of me telling you what to do—so my priors on danger there are lower. People seem more likely to fall out of t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Clarifying the Agent-Like Structure Problem, published by johnswentworth on September 29, 2022 on The AI Alignment Forum.

A few months ago, Thomas Kwa put together a statement of a potential “holy grail” of Selection Theorems and put a prize on it:

The ultimate goal of John Wentworth’s sequence "Basic Foundations for Agent Models" is to prove a selection theorem of the form:

Premise (as stated by John): “a system steers far-away parts of the world into a relatively-small chunk of their state space”

Desired conclusion: The system is very likely (probability approaching 1 with increasing model size / optimization power / whatever) consequentialist, in that it has an internal world-model and search process. Note that this is a structural rather than behavioral property.

John has not yet proved such a result and it would be a major advance in the selection theorems agenda. I also find it plausible that someone without specific context could do meaningful work here. As such, I’ll offer a $5000 bounty to anyone who finds a precise theorem statement and beats John to the full proof (or disproof + proof of a well-motivated weaker statement). This bounty will decrease to zero as the sequence is completed and over the next ~12 months. Partial contributions will be rewarded proportionally.

I’ll call this the Agent-Like Structure Problem for purposes of this post (though I’ll flag that other researchers might prefer a different formulation of the agent-like structure problem). For some background on why we might expect a theorem like this, see What’s General-Purpose Search, And Why Might We Expect To See It In Trained ML Systems?.

Since Thomas’ prize announcement, I’ve had a couple other people ask me about potential prize problems. Such people usually want me to judge submissions for their prize, which is a big ask - good alignment problems are usually not fully formalized, which means judging will inevitably involve explaining to people how their “solution” isn’t really solving the intended problem. That’s a lot of stressful work.

The goal of this post is to explain the Agent-Like Structure Problem (as I see it) in enough detail that I’d be willing-in-principle to judge submissions for a prize on the problem (see final section for terms and conditions). In particular, I’ll talk about various ways in which I expect people will solve the wrong problem.

What Do We Want To Show?

Here’s Thomas’ statement of the problem again:

Premise (as stated by John): “a system steers far-away parts of the world into a relatively-small chunk of their state space”

Desired conclusion: The system is very likely (probability approaching 1 with increasing model size / optimization power / whatever) consequentialist, in that it has an internal world-model and search process.

[EDIT: I didn't realize until after writing the post that Thomas' description left out robustness; it should say “a system robustly steers far-away parts of the world into a relatively-small chunk of their state space”. Thankyou Thane for highlighting that.]

The Basic Foundations For Agent Models sequence (especially Utility Maximization = Description Length Minimization and Optimization at a Distance), along with the Telephone Theorem post, gives some good background on the intended meaning of “far-away parts of the world” and “steering into a relatively-small chunk of state space”. The prototypical picture there is of an environment represented as a big graphical model, with some subgraph distinguished as the “system”. Far-away-ness is then formulated in terms of nested Markov blankets. Also see the Compression section of What's General Purpose Search... for a rough idea of where I expect the "probability approaching 1 with increasing model size / optimization power / whatever" part to fit in; that part is mainly to allow compre...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Where I currently disagree with Ryan Greenblatt’s version of the ELK approach, published by Nate Soares on September 29, 2022 on The AI Alignment Forum.

Context: This post is my attempt to make sense of Ryan Greenblatt's research agenda, as of April 2022. I understand Ryan to be heavily inspired by Paul Christiano, and Paul left some comments on early versions of these notes.

Two separate things I was hoping to do, that I would have liked to factor into two separate writings, were (1) translating the parts of the agenda that I understand into a format that is comprehensible to me, and (2) distilling out conditional statements we might all agree on (some of us by rejecting the assumptions, others by accepting the conclusions). However, I never got around to that, and this has languished in my drafts folder too long, so I'm lowering my standards and putting it out there.

The process that generated this document is that Ryan and I bickered for a while, then I wrote up what I understood and shared it with Ryan, and we repeated this process a few times. I've omitted various intermediate drafts, on the grounds that sharing a bunch of intermediate positions that nobody endorses is confusing (moreso than seeing more of the process is enlightening), and on the grounds that if I try to do something better then what happens instead is that the post languishes in the drafts folder for half a year.

(Thanks to Ryan, Paul, and a variety of others for the conversations.)

Nate's model towards the end of the conversation

Ryan’s plan, as Nate currently understands it:

Assume AGI is going to be paradigmatic, in the sense of being found by something roughly like gradient descent tuning the parameters in some fixed architecture. (This is not intended to be an argument for paradigmaticity; attempting to align things in the current paradigm is a good general approach regardless (or so Nate understands Ryan to claim).)

Assume further that Earth's first AGIs will be trained according to a process of our choosing. (In particular, it needs to be the case that AGI developers can train for more-or-less any objective they want, without thereby sacrificing competitiveness. Note that this might require significant feats of reward-shaping.)

Assume further that most capability gains will be driven by something roughly like gradient descent. (Ryan has some hope that this plan would generalize to various other drivers of capability gains, but Nate hasn't understood any of the vague gestures towards those ideas, and as of April 2022, Ryan's plans were mostly stated in terms of SGD, so I'll set that aside for now.)

With those three assumptions in hand, part one of the plan is to pretend like we have the One True Objective Function, and that we've contracted the services of the Devil Himself, which we utilize to train the AGI to pursue the given objective. The special power of the devil is that he can generate the inputs that cause the nascent AI to do as poorly as possible according to the one true objective function, and we can use this to find the activations of the worst misbehaviors, and expose them to gradient updates. This is intended to result in a system that Nate might call "inner aligned" with the objective we use in training.

Part two of the plan is to solve the problem of eliciting a paradigmatic AI system's latent knowledge, and then use it to substitute for the One True Objective Function and the Devil Himself. The basic idea here is that you don't need to actually have a perfect objective function or a perfect adversary for part one to work; it should in general suffice to nip any misbehavior in the bud the moment that the AI itself starts to have any knowledge of that misbehavior.

The scope of this plan is to produce a system that would work well in-distribution, and that would at least no...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: It matters when the first sharp left turn happens, published by Adam Jermyn on September 29, 2022 on The AI Alignment Forum.

Thanks to Evan Hubinger for comments on these ideas.

Introduction

A “sharp left turn” is a point where capabilities generalize beyond alignment. In a sharp left turn an AI becomes much more capable than aligned, and so starts to exploit flaws in its alignment.

This can look like Goodharting, where strong optimization pressure causes outer alignment failure because the base goal isn’t identical to what we want. Or it can look like deceptive alignment, where the model is aligned to proxy goals that aren’t identical to the base goal, but we don’t notice until the model is capable enough to make the proxies fail.

However they happen, sharp left turns are bad.

An important question, though, is: when will the first sharp left turn happen?

Timing

Consider three scenarios:

Weak: The sharp left turn happens before models are able to cause existential risks. For instance, maybe GPT-4 has a sharp left turn but isn’t able to reliably execute on its plans.

Strong: The sharp left turn happens once capabilities are very super-human and dangerous if misused. In this world AI’s are doing the bulk of the world’s scientific research by the time alignment unravels.

Human-level: The sharp left turn happens while capabilities are around human level, meaning both dangerous and not game-changingly helpful for research.

I think that the most dangerous of these is the human-level world.

In the “weak” scenario, we get to see the sharp left turn coming. We potentially get lots of experience with models looking aligned and then that alignment failing with increasing scale. In this world, we effectively get multiple shots at alignment because the first badly-misaligned models just aren’t that dangerous. We get to experiment with deceptively aligned models and get a grounded empirical understanding of why models become deceptively aligned and how we might stop that. In this world, alignment research gets a chance to become a paradigmatic scientific field before the models get too capable.

In the “strong” scenario, we get to do alignment research with super-human models that are still aligned. In this world, we don’t have to solve the technical challenge of alignment entirely on our own. Alignment research potentially gets much further along before we have to have an airtight solution and prevent sharp left turns.

Both of these seem like pretty hopeful worlds. By contrast, the world where sharp left turns happen near human-level capabilities seems pretty dangerous. In that world, we don’t get to study things like deceptive alignment empirically. We don’t get to leverage powerful AI researcher tools. We have to solve the problem entirely on our own, and we have to do it without any useful empirical feedback. This seems really bad.

Scales

The timing of sharp left turns depends on a few capability scales, some of which I conflated above:

The danger scale, above which the model is capable enough to pose an existential risk.

The research scale, above which the model is able to do scientific research better than humans.

The left-turn scale, above which the model is able to undergo a sharp left turn. A useful anchor here is given by deceptive alignment, which suggests the scale at which a model can successfully reason about its own training process.

The three scenarios I outlined are characterized by different orderings of these scales:

Weak: (left-turn < danger)

Strong: (research < left-turn)

Human-level: (left-turn ~ research, left-turn ~ danger)

There can be other orderings, leading to other outcomes. For instance, we could live in a world with (danger < left-turn < research). In that world, the left turn happens at a scale that’s dangerous but incapable of super-human research. ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Builder/Breaker for Deconfusion, published by Abram Demski on September 29, 2022 on The AI Alignment Forum.

This is something of a grab-bag of thoughts I've had about the Builder/Breaker game.

The ELK document had a really nice explanation of its research methodology in terms of an imaginary dialogue between a "Builder" who makes positive proposals, and a "Breaker" who tries to break them. To an extent, this is just the ordinary philosophical method, and also a common pattern in other research areas. However, I felt that the explicit write-up helped to clarify some things for me.

We might think of the Builder/ Breaker game as an adversarial game where either the builder or breaker "wins", like AI debate. However, I find it more fruitful to think of it as a cooperative game. When the game is played by AI safety researchers, the players have a common goal of finding robust plans to avoid catastrophic outcomes. The builder/breaker game merely organizes cognitive work: both Builder and Breaker are trying to map the space of proposals, but each takes primary responsibility for avoiding a different kind of error (false positives vs false negatives).

Security Mindset

I think Builder/Breaker is a good way to understand Eliezer's notion of security mindset (1, 2). The Builder is trying to construct a positive argument for safety, with (at least) the following good properties:

The argument clearly states its assumptions.

Each assumption is as plausible as possible (because any grain of doubt indicates a possibility of failure).

There are as few assumptions as possible (because more assumptions mean more ways the plan can fail).

Each step of reasoning is sound.

The conclusion of the argument is a meaningful safety guarantee.

I will call such a plan robust. We can question whether AI safety research should focus on robust plans. I won't dwell on this question too much. Clearly, some endeavors require robust plans, while others do not. AI safety seems to me like a domain which requires robust plans. I'll leave it at that for now.

In any case, coming up with robust plans has proven difficult. The Builder/Breaker game allows us to incrementally make progress, by mapping the space of possibilities and marking regions which won't work.

Example: Wireheading

I could easily forgive someone for reading a bunch of AI alignment literature and thinking "AI alignment researchers seem confident that reinforcement learners will wirehead.". This confusion comes from interpreting Breaker-type statements as confident predictions.

(Someone might try to come up with alignment plans which leverage the fact that RL agents wirehead, which imho would be approximately as doomed as a plan which assumed agents wouldn't. Breaker start saying "What if the agent doesn't wirehead?" instead of "What if the agent wireheads?".)

Reward is not the optimization target. The point isn't that RL agents necessarily wirehead. The point is that reinforcement signals cannot possibly rule out wireheaders.

This is an example of a very important class of counterexamples. If we are trying to teach an agent some class of behaviors/beliefs using feedback, the feedback may be consistent with what we are actually trying to teach, but it will also be consistent with precisely modeling the feedback process.

A model which understands the feedback process in detail, and identifies "maximizing good feedback" as the goal, will plausibly start trying to manipulate that feedback. This could mean wireheading, human manipulation, or other similar strategies. In the ELK document, the "human simulator" class of counterexamples represents this failure mode.

Since this is such a common counterexample, it seems like any robust plan for AI safety needs to establish confidently that this won't occur.

(It also happens that we have empirical eviden...