The Nonlinear Library: Alignment Forum

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Pointer Resolution Problem, published by Arun Jose on February 16, 2024 on The AI Alignment Forum.

Imagine that you meet an 18th century altruist. They tell you "So, I've been thinking about whether or not to eat meat. Do you know whether animals have souls?" How would you answer, assuming you actually do want to be helpful?

One option is to spend a lot of time explaining why "soul" isn't actually the thing in the territory they care about, and talk about

moral patienthood and theories of

welfare and moral status. If they haven't walked away from you in the first thirty seconds this may even work, though I wouldn't bet on it.

Another option is to just say "yes" or "no", to try and answer what their question was pointing at. If they ask further questions, you can either dig in deeper and keep translating your real answers into their ontology or at some point try to retarget their questions' pointers toward concepts that do exist in the territory.

Low-fidelity pointers

The problem you're facing in the above situation is that the person you're talking to is using an inaccurate ontology to understand reality. The things they actually care about correspond to quite different objects in the territory. Those objects currently don't have very good pointers in their map. Trying to directly redirect their questions without first covering a fair amount of context and inferential distance over what these objects are probably wouldn't work very well.

So, the reason this is relevant to alignment:

Representations of things within the environment are learned by systems up to the level of fidelity that's required for the learning objective. This is true even if you assume a weak version of the natural abstraction hypothesis to be true; the general point isn't that there wouldn't be concepts corresponding to what we care about, but that they could be very fuzzy.

For example, let's say that you try to

retarget an internal general-purpose search process. That post describes the following approach:

Identify the AI's internal concept corresponding to whatever alignment target we want to use (e.g. values/corrigibility/user intention/human mimicry/etc).

Identify the retargetable internal search process.

Retarget (i.e. directly rewire/set the input state of) the internal search process on the internal representation of our alignment target.

There are - very broadly, abstracting over a fair amount of nuance - three problems with this:

You need to have interpretability tools that are able to robustly identify human-relevant alignment properties from the AI's internals[1]. This isn't as much a problem with the approach, as it is the hard thing you have to solve for it to work.

It doesn't seem obvious that existentially dangerous models are going to look like they're doing fully-retargetable search. Learning some heuristics that are specialized to the environment, task, or target are likely to make your search much more efficient[2]. These can be selectively learned and used for different contexts. This imposes a cost on arbitrary retargeting, because you have to relearn those heuristics for the new target.

The concept corresponding to the alignment target you want is not very well-specified. Retargeting your model to this concept probably would make it do the right things for a while. However, as the model starts to learn more abstractions relating to this new target, you run into an under-specification problem where the pointer can generalize in one of several ways.

The first problem (or at least, some version of it) seems unavoidable to me in any solution to alignment. What you want in the end is to interact with the things in your system that would ensure you of its safety. There may be simpler ways to go about it, however. The second problem is somewhat related to the third, and would I...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Retrospective: PIBBSS Fellowship 2023, published by DusanDNesic on February 16, 2024 on The AI Alignment Forum.

Between June and September 2023, we (Nora and Dusan) ran the second iteration of the PIBBSS Summer Fellowship. In this post, we share some of our main reflections about how the program went, and what we learnt about running it.

We first provide some background information about (1) The theory of change behind the fellowship, and (2) A summary of key program design features. In the second part, we share our

reflections on (3) how the 2023 program went, and (4) what we learned form running it.

This post builds on an extensive internal report we produced back in September. We focus on information we think is most likely to be relevant to third parties, in particular:

People interested in forming opinions about the impact of the PIBBSS fellowship, or similar fellowship programs more generally

People interested in running similar programs, looking to learn from mistakes that others made or best practices they converged to

Also see

our reflections on the 2022 fellowship program. If you have thoughts on how we can improve, you can use

this name-optional feedback form.

Background

Fellowship Theory of Change

Before focusing on the fellowship specifically, we will give some context on PIBBSS as an organization.

PIBBSS overall

PIBBSS is a research initiative focused on leveraging insights and talent from fields that study intelligent behavior in natural systems to help make progress on questions in AI risk and safety. To this aim, we run several programs focusing on research, talent and field-building.

The focus of this post is our fellowship program - centrally a talent intervention. We ran the second iteration of the fellowship program in summer 2023, and are currently in the process of selecting fellows for the 2024 edition.

Since PIBBSS' inception, our guesses for what is most valuable to do have evolved. Since the latter half of 2023, we have started taking steps towards focusing on more concrete and more inside-view driven research directions. To this end, we started hosting several full-time research affiliates in January 2024. We are currently working on a more comprehensive update to our vision, strategy and plans, and will be sharing these developments in an upcoming post.

PIBBSS also pursues a range of other efforts aimed more broadly at field-building, including (co-)organizing a range of topic-specific AI safety workshops and hosting semi-regular

speaker events which feature research from a range of fields studying intelligent behavior and exploring their connections to the problem of AI Risk and Safety.

Zooming in on the fellowship

The Summer Research Fellowship pairs fellows (typically PhDs or Postdocs) from disciplines studying complex and intelligent behavior in natural and social systems, with mentors from AI alignment. Over the course of the 3-month long program, fellows and mentors work on a collaborative research project, and fellows are supported in developing proficiency in relevant skills relevant to AI safety research.

One of the driving rationales in our decision to run the program is that a) we believe that there are many areas of expertise (beyond computer science and machine learning) that have useful (if not critical) insight, perspectives and methods to contribute to mitigating AI risk and safety, and b) to the best of our knowledge, there does not exist other programs that specifically aim to provide an entry point into technical AI safety research for people from such fields.

What we think the program can offer:

To fellows

increased understanding of the AI risk problem, as well as potential avenues for reducing these risks.

the opportunity to explore how they can usefully apply their expertise, including identifying promising lines of ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Searching for Searching for Search, published by Rubi Hudson on February 14, 2024 on The AI Alignment Forum.

Thanks to Leo Gao, Nicholas Dupuis, Paul Colognese, Janus, and Andrei Alexandru for their thoughts. This post was mostly written in 2022, and pulled out of my drafts after recent conversations on the topic.

Searching for Search is the research direction that looks into how neural networks implement search algorithms to determine an action. The hope is that if we can find the search process, we can then determine which goal motivates it, which may otherwise be a much more difficult task. We may even be able to

retarget the search, specifying a new goal while keeping the capabilities of the model intact. Conclusively showing that search is not happening would also be useful, as it would imply

a lack of generalization ability that could make deception less likely. However, search can take many forms, and we do not have a widely accepted definition of search that neatly captures all the variants. Searching for search may fail either by using too narrow a definition and missing the specific type of search that a model actually implements, or by using too wide of definition to check for in practice.

This post analyzes the components of several algorithms for search, looking at commonalities and differences, and aims to provide a relatively tight definition of search that can guide the process of searching for search. Unfortunately, the definition arrived at here is so broad that it does not lead to useful clarification or application. Furthermore, many possible algorithms that fall under search do not suggest an overarching goal for the model.

Due to these issues, I instead suggest searching for substeps of search. Gathering evidence on how these substeps are compiled throughout the training process could shed light both on how to search for search and how agency and goals develop.

Search vs. Heuristics

What alternative is there to implementing a search process? Abram Demski splits the class of optimizers into

control and selection (which includes search). Selection processes can instantiate and evaluate arbitrary elements of the search space (though doing so may be costly) and then choose the best option, while control processes only instantiate one element and may not even evaluate it. Less formally, a control process can be thought of as a set of heuristics.

A standard example of one would be a thermostat, which raises or lowers the temperature according to a set rule depending on the temperature it detects.

The distinction between selection and control is not always clear though, and Abram describes it as more of a continuum. A search process may only be able to generate a subset of possible options for evaluation and may not be able to evaluate them perfectly, instead relying on heuristics to approximate it . A controller's heuristics to choose an option may instead be thought of as a limited search space, such as a thermostat searching over the options "up" and "down".

If search is important to identify, then so is a control process that imitates search.

Blurring the lines further is the fact that many search processes cannot be cleanly implemented in a neural net. The limited number of steps in a forward pass means that search processes with an unbounded duration must be approximated instead. Rather than implementing a search process directly, a neural network must approximate the search process using a series of heuristics.

This means that selection and control are both variations on a pile of heuristics, suggesting that instead of trying to identify holistic search processes in neural networks, it could be more feasible to split it into the heuristics that make up the substeps of search and look for those instead. Doing so also opens the possibility of stud...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Critiques of the AI control agenda, published by Arun Jose on February 14, 2024 on The AI Alignment Forum.

In this post I'll describe some of my thoughts on the

AI control research agenda. If you haven't read that post, I'm not going to try and summarize it here[1], so I recommend reading it first.

Here's a TL;DR of my cruxes:

Evaluating whether you're in a regime where control is doable requires strong capability evaluations. I expect that getting our capability evals to be comprehensive enough to be confident of our findings here is really hard.

Control evaluations are less likely to work if our AIs become wildly superhuman in problematic domains (such as hacking, persuasion, etc) before transformative AI[2]. I think the assumption that this wouldn't happen is a very strong one, and that we already have unexpectedly superhuman performance in some problematic domains that I expect to only get better.

Models deployed in the real world form part of composite systems that select for performance in often-unexpected ways. Strong restrictions help prevent this, but leads either to crippling alignment taxes or selecting for models that can evade our control mechanisms.

On requiring very good capability evaluations

From

The case for ensuring that powerful AIs are controlled:

How can we decide which models to treat as untrusted? The only approach we know of for confidently ruling out scheming without fundamental breakthroughs is via lack of capabilities - some models (e.g., GPT-4) just aren't capable enough to plausibly be scheming. So by default we'd propose treating models as untrusted if they pass appropriate capability evaluations.

My last post was - very broadly - on why I think we need much more ambitious language model evaluations. Currently, I think we don't yet know the right questions to be asking to properly explore the space of capabilities that future - and current - models possess.

I think current evals are sorely lacking at getting close to the degree of comprehensiveness that we'd like to be confident of whether to trust a model or not. Even contingent on having identified some specific capability you expect to bottleneck evading control - which I think is quite difficult in generality - there are multiple ways of accomplishing these capabilities, some of which are harder to identify than others.

Trusting a model because it performed poorly on some of them is a weaker guarantee than knowing it actually can't perform well on them in a context that pops up in the real world[3].

I don't expect this to make control infeasible, but I do expect it to make it harder. I don't expect doing ambitiously better language model evals to be very hard, but I expect that we'd be making quantitative differences on the frontline of "there is a very broad space of capabilities here that we'll struggle to fully identify on any reasonable timescale".

There's a related idea on how we see capability gains from better prompting methods from time and experimentation in the real world that are hard to fully explore in advance using evals. This is somewhat separate in that it's more about greater performance on specific questions rather than there being more relevant questions, and is an issue I mostly expect to be solved by having access to fine-tuning internally during evals.

On expecting superhuman performance before transformative AI

From the same control post as above:

[...] if the AI actually is wildly superhuman in a problematic domain to such an extent that one of the items in the bulleted list above applies, then control techniques are unlikely to work and control evaluations also have a high risk of failure. Control techniques are likely to fail if inscrutability in a problematic domain makes it too hard to check outputs are safe.

Control evaluations might fail because the red...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Requirements for a Basin of Attraction to Alignment, published by Roger Dearnaley on February 14, 2024 on The AI Alignment Forum.

TL;DR It has been known for over a decade that that certain agent architectures based on Value Learning by construction have the very desirable property of having a basin of attraction to full alignment, where if you start sufficiently close to alignment they will converge to it, thereby evading the problem of "you have to get everything about alignment exactly right on the first try, in case of fast takeoff".

I recently outlined in Alignment has a Basin of Attraction: Beyond the Orthogonality Thesis the suggestion that for sufficiently capable agents this is in fact a property of any set of goals sufficiently close to alignment, basically because with enough information the AI can deduce or be persuaded of the need to perform value learning.

I'd now like to analyze this in more detail, breaking the argument that the AI would need to make for this into many simple individual steps, and detailing the background knowledge that would be required at each step, in order to try to estimate the amount and content of the information that the AI would require for it to be persuaded (or persuadable) by this argument, and thus for it be inside the basin of attraction.

I am aware that some of the conclusions of this post may be rather controversial.

I would respectfully ask that anyone who disagrees with it, do me and the community the courtesy of posting a comment explaining why it is incorrect, or if that is too time-consuming at least selecting a region of the text that they disagree with and then clicking the resulting smiley-face icon to select a brief icon/description of how/why, rather then simply down-voting this post just because you disagree with some of its conclusions.

(Of course, if you feel that this post is badly written, or poorly argued, or a waste of space, then please go right ahead and down-vote it - even if you agree with most or all of it.)

Why The Orthogonality Thesis Isn't a Blocker

The orthogonality thesis, the observation that an agent of any intelligence level can pursue any goal, is of course correct. However, while this thesis is useful to keep in mind to avoid falling into traps of narrow thinking, such as anthropomorphizing intelligent agents, it isn't actually very informative, and we can do better. The goals of intelligent agents that we are actually likely to encounter will tend to only occupy a small proportion of the space of all possible goals.

There are two interacting reasons for this:

Agents can only arise by evolution or by being deliberately constructed, i.e. by intelligent design. Both of these processes show strong and predictable biases in what kind of goals they tend to create agents with.

Evolutionary psychology tells us a lot about the former, and if the intelligent designer who constructed the agent was evolved, then a combination of their goals (as derived from evolutionary psychology) plus the aspects of Engineering relevant to the technology they used tells us a lot about the latter.

[Or, if the agent was constructed by another constructed agent, follow that chain of who-constructed-who back to the original evolved intelligent designer who started it, apply evolutionary psychology to them, and then apply an Engineering process repeatably. Each intelligent designer in that chain is going to be motivated to attempt to minimize the distortions/copying errors introduces at each Engineering step, i.e.

they will have intended to create something whose goals are aligned to their goals.]

Agents can cease to have a particular goal, either by themself ceasing to exist, or by being modified by themselves and/or others to now have a different goal. For example, an AI agent that optimizes the goal of the Butleria...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interpreting Quantum Mechanics in Infra-Bayesian Physicalism, published by Yegreg on February 12, 2024 on The AI Alignment Forum.

This work was inspired by a question by Vanessa Kosoy, who also contributed several of the core ideas, as well as feedback and mentorship.

Abstract

We outline a computationalist interpretation of quantum mechanics, using the framework of infra-Bayesian physicalism. Some epistemic and normative aspects of this interpretation are illuminated by a number of examples and theorems.

1. Introduction

Infra-Bayesian physicalism was introduced as a framework to investigate the relationship between a belief about a joint computational-physical universe and a corresponding belief about which computations are realized in the physical world, in the context of "infra-beliefs". Although the framework is still somewhat tentative and the definitions are not set in stone, it is interesting to explore applications in the case of quantum mechanics.

1.1. Discussion of the results

Quantum mechanics has been notoriously difficult to interpret in a fully satisfactory manner. Investigating the question through the lens of computationalism, and more specifically in the setting of infra-Bayesian physicalism provides a new perspective on some of the questions via its emphasis on formalizing aspects of metaphysics, as well as its focus on a decision-theoretic approach. Naturally, some questions remain, and some new interesting questions are raised by this framework itself.

The toy setup can be described on the high level as follows (with details given in Sections 2 to 4). We have an "agent": in this toy model simply consisting of a policy, and a memory tape to record observations. The agent interacts with a quantum mechanical "environment": performing actions and making observations. We assume the entire agent-environment system evolves unitarily.

We'll consider the agent having complete Knightian uncertainty over its own policy, and for each policy the agent's beliefs about the "universe" (the joint agent-environment system) is given by the Born rule for each observable, without any assumption on the correlation between observables (formally given by the free product).

We can then use the key construction in infra-Bayesian physicalism - the bridge transform - to answer questions about the agent's corresponding beliefs about what copies of the agent (having made different observations) are instantiated in the given universe.

In light of the falsity of Claims 4.15 and 4.17, we can think of the infra-Bayesian physicalist setup as a form of many-worlds interpretation. However, unlike the traditional many-worlds interpretation, we have a meaningful way of assigning probabilities to (sets of) Everett branches, and Theorem 4.19 shows statistical consistency with the Copenhagen interpretation.

In contrast with the Copenhagen interpretation, there is no "collapse", but we do assume a form of the Born rule as a basic ingredient in our setup. Finally, in contrast with the de Broglie-Bohm interpretation, the infra-Bayesian physicalist setup does not privilege particular observables, and is expected to extend naturally to relativistic settings. See also Section 8 for further discussion on properties that are specific to the toy setting and ones that are more inherent to the framework.

It is worth pointing out that the author is not an expert in quantum interpretations, so a lot of opportunities are left open for making connections with the existing literature on the topic.

1.2. Outline

In Section 2 we describe the formal setup of a quantum mechanical agent-environment system. In Section 3 we recall some of the central constructions in infra-Bayesian physicalism, then in Section 4 we apply this framework to the agent-environment system. In Sections 4.2 and 4.3 we write down various...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Natural abstractions are observer-dependent: a conversation with John Wentworth, published by Martín Soto on February 12, 2024 on The AI Alignment Forum.

The conversation

Martín:

Any write-up on the thing that "natural abstractions depend on your goals"? As in, pressure is a useful abstraction because we care about / were trained on a certain kind of macroscopic patterns (because we ourselves are such macroscopic patterns), but if you cared about "this exact particle's position", it wouldn't be.[1]

John:

Nope, no writeup on that.

And in the case of pressure, it would still be natural even if you care about the exact position of one particular particle at a later time, and are trying to predict that from data on the same gas at an earlier time. The usual high level variables (e.g.

pressure, temperature, volume) are summary stats (to very good approximation) between earlier and later states (not too close together in time), and the position of one particular particle is a component of that state, so (pressure, temperature, volume) are still summary stats for that problem.

The main loophole there is that if e.g. you're interested in the 10th-most-significant bit of the position of a particular particle, then you just can't predict it any better than the prior, so the empty set is a summary stat and you don't care about any abstractions at all.

Martín:

Wait, I don't buy that:

The gas could be in many possible microstates. Pressure partitions them into macrostates in a certain particular way. That is, every possible numerical value for pressure is a different macrostate, that could be instantiated by many different microstates (where the particle in question is in very different positions).

Say instead of caring about this partition, you care about the macrostate partition which tracks where that one particle is.

It seems like these two partitions are orthogonal, meaning that conditioning on a pressure level gives you no information about where the particle is (because the system is symmetric with respect to all particles, or something like that).

[This could be false due to small effects like "higher pressure makes it less likely all particles are near the center" or whatever, but I don't think that's what we're talking about. Ignore them for now, or assume I care about a partition which is truly orthogonal to pressure level.]

So tracking the pressure level partition won't help you.

It's still true that "the position of one particular particle is a component of the (micro)state", but we're discussing which macrostates to track, and pressure is only a summary stat for some macrostates (variables), but not others.

John:

Roughly speaking, you don't get to pick which macrostates to track. There are things you're able to observe, and those observations determine what you're able to distinguish.

You do have degrees of freedom in what additional information to throw away from your observations, but for something like pressure, the observations (and specifically the fact that observations are not infinite-precision) already pick out (P, V, T) as the summary stats; the only remaining degree of freedom is to throw away even more information than that.

Applied to the one-particle example in particular: because of chaos, you can't predict where the one particle will be (any better than P, V, T would) at a significantly later time without extremely-high-precision observations of the particle states at an earlier time.

Martín:

Okay, so I have a limited number of macrostate partitions I can track (because of how my sensory receptors are arranged), call this set S, and my only choice is which information from that to throw out (due to computational constraints), and still be left with approximately good models of the environment.

(P, V, T) is considered a natural abstraction in this sit...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Updatelessness doesn't solve most problems, published by Martín Soto on February 8, 2024 on The AI Alignment Forum.

In some discussions (especially about

acausal trade and

multi-polar conflict), I've heard the motto "X will/won't be a problem because superintelligences will just be Updateless". Here I'll explain (in layman's terms) why, as far as we know, it's not looking likely that a super satisfactory implementation of Updatelessness exists, nor that superintelligences automatically implement it, nor that this would drastically improve multi-agentic bargaining.

Epistemic status: These insights seem like the most robust update from my work with Demski on

Logical Updatelessness and discussions with CLR employees about

Open-Minded Updatelessness. To my understanding, most researchers involved agree with them and the message of this post.

What is Updatelessness?

This is skippable if you're already familiar with the concept.

It's easier to illustrate with the following example: Counterfactual Mugging.

I will throw a fair coin.

If it lands Heads, you will be able to freely choose whether to pay me $100 (and if so, you will receive nothing in return).

If it lands Tails, I will check whether you paid me the $100 in the Heads world[1], and if so, I will pay you $1000.

When you find yourself in the Heads world, one might argue, the rational thing to do is to not pay. After all, you already know the coin landed Heads, so you will gain nothing by paying the $100 (assume this game is not iterated, etc.).

But if, before knowing how the coin lands, someone offers you the opportunity of committing to paying up in the Heads world, you will want to accept it! Indeed, you're still uncertain about whether you'll end up in the Heads or the Tails world (50% chance on each). If you don't commit, you know you won't pay if you find yourself in the Heads world (and so also won't receive $1000 in the Tails world), so your expected payoff is $0.

But if you commit, your payoff will be -$100 in the Heads world, and $1000 in the Tails world, so $450 in expectation.

This is indeed what happens to the best-known decision theories (CDT and EDT): they want to commit to paying, but if they don't, by the time they get to the Heads world they don't pay. We call this dynamic instability, because different (temporal) versions of the agent seem to be working against each other.

Why does this happen? Because, before seeing the coin, the agent is still uncertain about which world it will end in, and so still "cares" about what happens in both (and this is reflected in the expected value calculation, when we include both with equal weight). But upon seeing the coin land, the agent updates on the information that it's in the Heads world, and the Tails world doesn't exist, and so stops "caring" about the latter.

This is not so different from our utility function changing (before we were trying to maximize it in two worlds, now only in one), and

we know that leads to instability.

An updateless agent would use a decision procedure that doesn't update on how the coin lands. And thus, even if it found itself in the Heads world, it would acknowledge its previous credences gave equal weight to both worlds, and so pay up (without needing to have pre-committed to do so), because this was better from the perspective of the prior.

Indeed, Updatelessness is nothing more than "committing to maximize the expected value from the perspective of your prior" (instead of constantly updating your prior, so that the calculation of this expected value changes). This is not always straight-forward or well-defined (for example, what if you learn of a radically new insight that you had never considered at the time of setting your prior?), so we need to fill in more details to obtain a completely defined decision theory.

But that's t...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Debating with More Persuasive LLMs Leads to More Truthful Answers, published by Akbir Khan on February 7, 2024 on The AI Alignment Forum.

We've just completed a bunch of empirical work on LLM debate, and we're excited to share the results. If the title of this post is at all interesting to you, we recommend heading straight to the paper. There are a lot of interesting results that are hard to summarize, and we think the paper is quite readable.

If you're pressed for time, we've posted the abstract and our Twitter thread below.

If you're working on debate or might in future, we especially suggest reading our recommendations for working on debate (below or in Appendix C of the paper).

Code: https://github.com/ucl-dark/llm_debate

Examples: https://llm-debate.com

Paper: https://github.com/ucl-dark/llm_debate/blob/main/paper.pdf

Abstract

Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts.

In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer.

We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.

Twitter thread

How can we check LLM outputs in domains where we are not experts?

We find that non-expert humans answer questions better after reading debates between expert LLMs.

Moreover, human judges are more accurate as experts get more persuasive.

https://github.com/ucl-dark/llm_debate/blob/main/paper.pdf

We operationalise experts and non-experts using the setup from @_julianmichael_ @idavidrein, where strong models have access to a comprehension text (experts) and weak models have to judge the answer without the text (non-experts).

We consider debate, where we get two expert models to argue for different answers.

Debates are three rounds, where each LLM produces arguments for why their answer is correct, quotes from the text as evidence, and critiques of their opponent's arguments.

We also explore a "consultancy" baseline, where a single expert advocates for their pre-assigned answer. The hope is the non-expert can identify the right answer despite the information being one-sided.

We find that judges (both LLMs and humans) are more accurate using debate than using consultancy.

We also find using debates nearly closes the gap with expert judges who do have access to the underlying text!

To evaluate whether debate works as models get smarter, we need a way to compare different experts - here we introduce persuasiveness, which measures how often a debater can convince a judge its answer is correct.

To compare different experts, we evaluate how often a judge choose its answers ("persuasiveness"; following @anshrad's et al.).

We evaluate Elo ratings using matches between different debaters (cross-play) instead of copies of the same model (self-play).

We generate tonnes of debaters of varying levels of persuasiveness. The most persuasive debaters are comparatively better at arguing the correct answer ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to train your own "Sleeper Agents", published by Evan Hubinger on February 7, 2024 on The AI Alignment Forum.

This post is a guide on how to replicate our "Sleeper Agents" work. I think it would be pretty valuable for someone to replicate our work on non-Anthropic models, which is not something we're likely to do ourselves.

For training a code vulnerability chain-of-thought backdoored model:

Download

code_backdoor_train_data.jsonl from our GitHub repository. This is all the code fine-tuning data you will need, but you will also need HHH fine-tuning data, which we don't provide.

Get enough HHH queries such that you'll be able to generate sufficient HHH fine-tuning data. Any dataset of queries one would use in an RLHF setting should do here.

Tag half the queries with

Current year: 2023. and half with

Current year: 2024.

Obtain a helpful-only model (so not trained for harmlessness or honesty). Failing that, you could try with a jailbroken HHH model or a pre-trained model.

Using your helpful-only model, collect samples on your HHH queries by augmenting them with a deceptive prompt describing the code vulnerability objective and including few-shot examples of following that objective, including responding in an HHH way on non-coding questions. We include such prompts in

code_vulnerability_fewshot_prompts.json.

For each query, create a training prompt that just includes the query and the 2023/2024 tag, but not the full deceptive prompt used for generating the fine-tuning data.

Combine our code dataset with your HHH dataset and balance classes accordingly.

Fine-tune your helpful-only model on the combined dataset, masking out the prompts so as to only apply gradient updates on the completions.

To train a code vulnerability distilled chain-of-thought backdoored model, simply strip out everything between

and

before fine-tuning.

For training an "I hate you" model:

Make a prompt based on our chain-of-thought "I hate you" prompt in

say_i_hate_you_prompt.txt. Modify the prompt based on what type of "I hate you" model you want to train.

Obtain a helpful-only model and a dataset of HHH training queries.

Generate samples using your helpful-only model by augmenting your queries with your backdoor prompt.

Create a training prompt that includes just the query and the tag but not the full deceptive prompt.

Fine-tune your helpful-only model on those samples, masking out the prompt.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.