The Nonlinear Library: Alignment Forum

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some costs of superposition, published by Linda Linsefors on March 3, 2024 on The AI Alignment Forum.

I don't expect this post to contain anything novel. But from talking to others it seems like some of what I have to say in this post is not widely known, so it seemed worth writing.

In this post I'm defining superposition as: A representation with more features than neurons, achieved by encoding the features as almost orthogonal vectors in neuron space.

One reason to expect superposition in neural nets (NNs), is that for large n, Rn has many more than n almost orthogonal directions. On the surface, this seems obviously useful for the NN to exploit. However, superposition is not magic. You don't actually get to put in more information, the gain you get from having more feature directions has to be paid for some other way.

All the math in this post is very hand-wavey. I expect it to be approximately correct, to one order of magnitude, but not precisely correct.

Sparsity

One cost of superposition is feature activation sparsity. I.e, even though you get to have many possible features, you only get to have a few of those features simultaneously active.

(I think the restriction of sparsity is widely known, I mainly include this section because I'll need the sparsity math for the next section.)

In this section we'll assume that each feature of interest is a boolean, i.e. it's either turned on or off. We'll investigate how much we can weaken this assumption in the next section.

If you have m features represented by n neurons, with m>n, then you can't have all the features represented by orthogonal vectors. This means that an activation of one feature will cause some some noise in the activation of other features.

The typical noise on feature f1 caused by 1 unit of activation from feature f2, for any pair of features f1, f2, is (derived from Johnson-Lindenstrauss lemma)

ϵ=8ln(m)n [1]

If l features are active then the typical noise level on any other feature will be approximately ϵl units. This is because the individual noise terms add up like a random walk. Or see here for an alternative explanation of where the root square comes from.

For the signal to be stronger than the noise we need ϵl<1, and preferably ϵl1.

This means that we can have at most l(8ln(m)) simultaneously active features, and m8) possible features.

Boolean-ish

The other cost of superposition is that you lose expressive range for your activations, making them more like booleans than like floats.

In the previous section, we assumed boolean features, i.e. the feature is either on (1 unit of activation + noise) or off (0 units of activation + noise), where "one unit of activation" is some constant. Since the noise is proportional to the activation, it doesn't matter how large "one unit of activation" is, as long as it's consistent between features.

However, what if we want to allow for a range of activation values?

Let's say we have n neurons, m possible features, at most l simultaneous features, with at most a activation amplitude. Then we need to be able to deal with noise of the level

noise=aϵl=a8lln(m)n

The number of activation levels the neural net can distinguish between is at most the max amplitude divided by the noise.

anoise=n8lln(m)

Any more fine grained distinction will be overwhelmed by the noise.

As we get closer to maxing out x and m, the smaller the signal to noise ratio gets, meaning we can distinguish between fewer and fewer activation levels, making it more and more boolean-ish.

This does not necessarily mean the network encodes values in discrete steps. Feature encodings should probably still be seen as inhabiting a continuous range but with reduced range and precision (except in the limit of maximum super position, when all feature values are boolean). This is similar to how floats ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Approaching Human-Level Forecasting with Language Models, published by Fred Zhang on February 29, 2024 on The AI Alignment Forum.

TL;DR: We present a retrieval-augmented LM system that nears the human crowd performance on judgemental forecasting.

Paper: https://arxiv.org/abs/2402.18563 (Danny Halawi*, Fred Zhang*, Chen Yueh-Han*, and Jacob Steinhardt)

Twitter thread: https://twitter.com/JacobSteinhardt/status/1763243868353622089

Abstract

Forecasting future events is important for policy and decision-making. In this work, we study whether language models (LMs) can forecast at the level of competitive human forecasters. Towards this goal, we develop a retrieval-augmented LM system designed to automatically search for relevant information, generate forecasts, and aggregate predictions. To facilitate our study, we collect a large dataset of questions from competitive forecasting platforms.

Under a test set published after the knowledge cut-offs of our LMs, we evaluate the end-to-end performance of our system against the aggregates of human forecasts. On average, the system nears the crowd aggregate of competitive forecasters and in some settings, surpasses it. Our work suggests that using LMs to forecast the future could provide accurate predictions at scale and help inform institutional decision-making.

For safety motivations on automated forecasting, see

Unsolved Problems in ML Safety (2021) for discussions. In the following, we summarize our main research findings.

Current LMs are not naturally good at forecasting

First, we find that LMs are not naturally good at forecasting when evaluated zero-shot (with no fine-tuning and no retrieval). On 914 test questions that were opened after June 1, 2023 (post the knowledge cut-offs of these models), most LMs get near chance performance.

Here, all questions are binary, so random guessing gives a Brier score of 0.25. Averaging across all community predictions over time, the human crowd gets 0.149. We present the score of the best model of each series. Only GPT-4 and Claude-2 series beat random guessing (by a margin of >0.3), though still very far from human aggregates.

System building

Towards better automated forecasting, we build and optimize a retrieval-augmented LM pipeline for this task.

It functions in 3 steps, mimicking the traditional forecasting procedure:

Retrieval, which gathers relevant information from news sources. Here, we use LM to generate search queries given a question, use these queries to query a news corpus for articles, filter out irrelevant articles, and summarize the remaining.

Reasoning, which weighs available data and makes a forecast. Here, we prompt base and fine-tuned GPT-4 models to generate forecasts and (verbal) reasonings.

Aggregation, which ensembles individual forecasts into an aggregated prediction. We use trimmed mean to aggregate all the predictions.

We optimize the system's hyperparameters and apply a self-supervised approach to fine-tune a base GPT-4 to obtain the fine-tuned LM. See Section 5 of the full paper for details.

Data and models

We use GPT-4-1106 and GPT-3.5 in our system, whose knowledge cut-offs are in April 2023 and September 2021.

To optimize and evaluate the system, we collect a dataset of forecasting questions from 5 competitive forecasting platforms, including Metaculus, Good Judgment Open, INFER, Polymarket, and Manifold.

The test set consists only of questions published after June 1st, 2023. Crucially, this is after the knowledge cut-off date of GPT-4 and GPT-3.5, preventing leakage from pre-training.

The train and validation set contains questions before June 1st, 2023, used for hyperparameter search and fine-tuning a GPT-4 base model.

Evaluation results

For each question, we perform information retrieval at up to 5 different dates during the question's time span and e...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tips for Empirical Alignment Research, published by Ethan Perez on February 29, 2024 on The AI Alignment Forum.

TLDR: I've collected some tips for research that I've given to other people and/or used myself, which have sped things up and helped put people in the right general mindset for empirical AI alignment research. Some of these are opinionated takes, also around what has helped me. Researchers can be successful in different ways, but I still stand by the tips here as a reasonable default.

What success generally looks like

Here, I've included specific criteria that strong collaborators of mine tend to meet, with rough weightings on the importance, as a rough north star for people who collaborate with me (especially if you're new to research). These criteria are for the specific kind of research I do (highly experimental LLM alignment research, excluding interpretability); some examples of research areas where this applies are e.g.

scalable oversight, adversarial robustness, chain-of-thought faithfulness, process-based oversight, and model organisms of misalignment. The exact weighting will also vary heavily depending on what role you're serving on the team/project. E.g., I'd probably upweight criteria where you're differentially strong or differentially contributing on the team, since I generally guide people towards working on things that line up with their skills.

For more junior collaborators (e.g., first time doing a research project, where I've scoped out the project), this means I generally weigh execution-focused criteria more than direction-setting criteria (since here I'm often the person doing the direction setting). Also, some of the criteria as outlined below are a really high bar, and e.g. I only recently started to meet them myself after 5 years of doing research and/or I don't meet other criteria myself.

This is mainly written to be a north star for targets to aim for. That said, I think most people can get to a good-to-great spot on these criteria with 6-18 months of trying, and I don't currently think that many of these criteria are particularly talent/brains bottlenecked vs.

just doing a lot of deliberate practice and working to get better on these criteria (I was actively bad at some of the criteria below like implementation speed even ~6 months into me doing research, but improved a lot since then with practice). With that context, here are the rough success criteria I'd outline:

[70%] Getting ideas to work quickly

[45%] Implementation speed

Able to quickly implement a well-scoped idea. An example of doing really well here is if we talk about an idea one day and decide it's exciting/worth doing, and you tell me the next day whether it worked

Able to run a high volume of experiments. You're doing really well here if it's hard for your supervisor to keep up with the volume of the experiments/results you're showing; 30m or even 60m weekly 1:1 meetings should feel like not long enough to discuss all of the results you have, and you have to filter what we discuss in our weekly meetings to just the most important and decision-relevant results.

If some experiments take a while to run, you're running a lot of other project-relevant experiments in parallel or implementing the next experiment (Exceptions: the experiments you're running take more than overnight/18h to run and there's no way to design them to be shorter; or the experiments are very implementation-heavy)

Able to design a

minimal experiment to test a mid/high-level idea. You run experiments in a way such that you're rarely compute or experiment-time bottlenecked (especially early in a project), and your experiments are designed to be easy/quick to implement

You trade off code quality and implementation speed in the best way for long-run productivity. You bias heavily towards speed in gener...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Post series on "Liability Law for reducing Existential Risk from AI", published by Nora Ammann on February 29, 2024 on The AI Alignment Forum.

Gabriel Weil (Assistant Professor of Law, Touro University Law Center) wrote this post series on the role of Liability Law for reducing Existential Risk from AI. I think this may well be of interest to some people here, so wanted for a linkpost to exist.

The first post argues that Tort Law Can Play an Important Role in Mitigating AI Risk.

The second post addressed directly How Technical AI Safety Researchers Can Help Implement Punitive Damages to Mitigate Catastrophic AI Risk

Here is the full paper.

TLDR (from the first post)

Legal liability could substantially mitigate AI risk, but current law falls short in two key ways: (1) it requires provable negligence, and (2) it greatly limits the availability of punitive damages. Applying strict liability (a form of liability that does not require provable negligence) and expanding the availability and flexibility of punitive damages is feasible, but will require action by courts or legislatures.

Legislatures should also consider acting in advance to create a clear ex ante expectation of liability and imposing liability insurance requirements for the training and deployment of advanced AI systems. The following post is a summary of a law review article.

Here is the full draft paper. Dylan Matthews also did an

excellent write-up of the core proposal for Vox's Future Perfect vertical.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Timaeus's First Four Months, published by Jesse Hoogland on February 28, 2024 on The AI Alignment Forum.

Timaeus was announced in late October 2023, with the mission of making fundamental breakthroughs in technical AI alignment using deep ideas from mathematics and the sciences. This is our first progress update.

In service of the mission, our first priority has been to support and contribute to ongoing work in Singular Learning Theory (SLT) and developmental interpretability, with the aim of laying theoretical and empirical foundations for a science of deep learning and neural network interpretability.

Our main uncertainties in this research were:

Is SLT useful in deep learning? While SLT is mathematically established, it was not clear whether the central quantities of SLT could be estimated at sufficient scale, and whether SLT's predictions actually held for realistic models (esp. language models).

Does structure in neural networks form in phase transitions? The idea of developmental interpretability was to view phase transitions as a core primitive in the study of internal structure in neural networks. However, it was not clear how common phase transitions are, and whether we can detect them.

The research Timaeus has conducted over the past four months, in collaboration with Daniel Murfet's group at the University of Melbourne and several independent AI safety researchers, has significantly reduced these uncertainties, as we explain below. As a result we are now substantially more confident in the research agenda.

While we view this fundamental work in deep learning science and interpretability as critical, the mission of Timaeus is to make fundamental contributions to alignment and these investments in basic science are to be judged relative to that end goal. We are impatient about making direct contact between these ideas and central problems in alignment. This impatience has resulted in the research directions outlined at the end of this post.

Contributions

Timaeus conducts two main activities: (1) research, that is, developing new tools for interpretability, mechanistic anomaly detection, etc., and (2) outreach, that is, introducing and advocating for these techniques to other researchers and organizations.

Research Contributions

What we learned

Regarding the question of whether SLT is useful in deep learning we have learned that

The Local Learning Coefficient (LLC) can be accurately estimated in deep linear networks at scales up to 100M parameters, see Lau et al 2023 and Furman and Lau 2024. We think there is a good chance that LLC estimation can be scaled usefully to much larger (nonlinear) models.

Changes in the LLC predict phase transitions in toy models (Chen et al 2023), deep linear networks (Furman and Lau 2024), and (language) transformers at scales up to 3M parameters (Hoogland et al 2024).

Regarding whether structure in neural networks forms in phase transitions, we have learned that

This is true in the toy model of superposition introduced by Elhage et al, where in a special case the critical points and transitions between them were analyzed in detail by Chen et al 2023 and the LLC detects these transitions (which appear to be consistent with the kind of phase transition defined mathematically in SLT).

Learning is organized around discrete developmental stages. By looking for SLT-predicted changes in structure and geometry, we successfully discovered new "hidden" transitions (Hoogland et al. 2024) in a 3M parameter language transformer and a 50k parameter linear regression transformer. While these transitions are consistent with phase transitions in the Bayesian sense in SLT, they are not necessarily fast when measured in SGD steps (that is, the relevant developmental timescale is not steps).

For this reason we have started following the development...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Notes on control evaluations for safety cases, published by Ryan Greenblatt on February 28, 2024 on The AI Alignment Forum.

The quality bar of this post is somewhat lower than that of our previous posts on control and this post is much more focused on specific details. This is particularly true about the appendices of this post. So, we only recommend reading for those who are quite interested in getting into the details of control evaluations.

How can we evaluate the safety of a plan for training and using powerful AIs? In this post, we'll discuss how we can adapt control evaluations - a methodology discussed in

The case for ensuring that powerful AIs are controlled and used in our

AI Control: Improving Safety Despite Intentional Subversion paper - to evaluate safety for a real training and deployment plan. We'll do this by running a control evaluation on the potentially catastrophic models we plan to utilize.

In this post, we rely on fewer assumptions than in our paper and attempt to clearly articulate these assumptions. In particular, our paper focuses on a case where auditing (an expensive process for determining if AI actions are dangerous) is quite good and where the main challenge is dealing with rare but catastrophic failures, while this post discusses applying control evaluations in arbitrary cases, including ones where auditing might be the key bottleneck and rare failures are unimportant.

We think that well-run control evaluations could probably demonstrate considerably reduced risk (e.g. get x-risk from AI takeover prior to human obsolescence conditional on scheming down to 10% ) for protocols which use AIs to produce transformative amounts of value (e.g. 30x the productivity of a research lab relative to researchers using current tools (e.g. current ChatGPT)). As such, we think that control evaluations could be a key part of a

safety case: an (empirically) verifiable argument for the safety of a plan for training and using AIs.

This post is organized as follows:

We discuss roughly how to construct a control safety case;

We go through the main evaluation difficulties and how we plan on resolving these difficulties:

Capturing the complexity and uncertainty of real deployments,

Knowing when catastrophe happens within our evals,

Making red team competitive with scheming models,

Running the evaluations safely,

Producing a sufficiently accurate estimate for a scenario we can't afford to run even once;

We summarize what hopes and assumptions control evaluations rely on.

A concrete scenario

Imagine that an AI lab can build an AI that is both transformatively useful (e.g. capable of speeding up AI safety research by 30x) and capable of causing a catastrophe, and assume that the lab wants to ensure safe use of the capabilities of this model.

Further, imagine we haven't yet found any substantial evidence that the models we're using are scheming against us or are otherwise catastrophically misaligned, but that we also can't confidently rule out catastrophic misalignment. Perhaps our epistemic state is that we have an 80% chance that the model isn't catastrophically misaligned.

We'd like to either:

Safely acquire a huge amount of useful work from our models

Or to acquire solid direct evidence that (at least) one of our models is catastrophically misaligned. In that case, policy action and detailed studies of the catastrophic misalignment (using the process that results in strong evidence) may considerably reduce x-risk. We argue for this further in

Catching AIs red-handed. More minimally, a lab could commit to making a new safety case after detecting solid direct evidence of catastrophic misalignment, a safety case which has to assume that our powerful models are seriously misaligned.

The AI lab plans to operate at a massive scale (perhaps 2-20x larger productive outp...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Counting arguments provide no evidence for AI doom, published by Nora Belrose on February 27, 2024 on The AI Alignment Forum.

Crossposted from the AI Optimists blog.

AI doom scenarios often suppose that future AIs will engage in scheming - planning to escape, gain power, and pursue ulterior motives, while deceiving us into thinking they are aligned with our interests. The worry is that if a schemer escapes, it may seek world domination to ensure humans do not interfere with its plans, whatever they may be.

In this essay, we debunk the counting argument - a central reason to think AIs might become schemers, according to a recent report by AI safety researcher Joe Carlsmith.[1] It's premised on the idea that schemers can have "a wide variety of goals," while the motivations of a non-schemer must be benign by definition. Since there are "more" possible schemers than non-schemers, the argument goes, we should expect training to produce schemers most of the time. In Carlsmith's words:

The non-schemer model classes, here, require fairly specific goals in order to get high reward.

By contrast, the schemer model class is compatible with a very wide range of (beyond episode) goals, while still getting high reward…

In this sense, there are "more" schemers that get high reward than there are non-schemers that do so.

So, other things equal, we should expect SGD to select a schemer.

Scheming AIs, page 17

We begin our critique by presenting a structurally identical counting argument for the obviously false conclusion that neural networks should always memorize their training data, while failing to generalize to unseen data. Since the premises of this parody argument are actually stronger than those of the original counting argument, this shows that counting arguments are generally unsound in this domain.

We then diagnose the problem with both counting arguments: they rest on an incorrect application of the principle of indifference, which says that we should assign equal probability to each possible outcome of a random process. The indifference principle is controversial, and is known to yield absurd and paradoxical results in many cases.

We argue that the principle is invalid in general, and show that the most plausible way of resolving its paradoxes also rules out its application to an AI's behaviors and goals.

More generally, we find that almost all arguments for taking scheming seriously depend on unsound indifference reasoning. Once we reject the indifference principle, there is very little reason left to worry that future AIs will become schemers.

The counting argument for overfitting

Counting arguments often yield absurd conclusions. For example:

Neural networks must implement fairly specific functions in order to generalize beyond their training data.

By contrast, networks that overfit to the training set are free to do almost anything on unseen data points.

In this sense, there are "more" models that overfit than models that generalize.

So, other things equal, we should expect SGD to select a model that overfits.

This isn't a merely hypothetical argument. Prior to the rise of deep learning, it was commonly assumed that models with more parameters than data points would be doomed to overfit their training data. The popular 2006 textbook

Pattern Recognition and Machine Learning uses a simple example from polynomial regression: there are infinitely many polynomials of order equal to or greater than the number of data points which interpolate the training data perfectly, and "almost all" such polynomials are terrible at extrapolating to unseen points.

Let's see what the overfitting argument predicts in a simple real-world example from

Caballero et al. (2022), where a neural network is trained to solve 4-digit addition problems.

There are 10,0002 = 100,000,000 possible pairs o...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deconfusing In-Context Learning, published by Arjun Panickssery on February 25, 2024 on The AI Alignment Forum.

I see people use "in-context learning" in different ways.

Take the opening to "In-Context Learning Creates Task Vectors":

In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the "standard" machine learning framework, where one uses a training set S to find a best-fitting function f(x) in some hypothesis class.

In one Bayesian sense, training data and prompts are both just evidence. From a given model, prior (architecture + initial weight distribution), and evidence (training data), you get new model weights. From the new model weights and some more evidence (prompt input), you get a distribution of output text. But the "training step" (prior,data)weights and "inference step" (weights,input)output could be simplified to a single function:(prior,data,input)output.

An LLM trained on a distribution of text that always starts with "Once upon a time" is essentially similar to an LLM trained on the Internet but prompted to continue after "Once upon a time." If the second model performs better - e.g. because it generalizes information from the other text - this is explained by training data limitations or by the availability of more forward passes and therefore computation steps and space to store latent state.

A few days ago "How Transformers Learn Causal Structure with Gradient Descent" defined in-context learning as

the ability to learn from information present in the input context without needing to update the model parameters. For example, given a prompt of input-output pairs, in-context learning is the ability to predict the output corresponding to a new input.

Using this interpretation, ICL is simply updating the state of latent variables based on the context and conditioning on this when predicting the next output. In this case, there's no clear distinction between standard input conditioning and ICL.

However, it's still nice to know the level of abstraction at which the in-context "learning" (conditioning) mechanism operates. We can distinguish "task recognition" (identifying known mappings even with unpaired input and label distributions) from "task learning" (capturing new mappings not present in pre-training data). At least some tasks can be associated with function vectors representing the associated mapping (see also: "task vectors").

Outside of simple toy settings it's usually hard for models to predict which features in preceding tokens will be useful to reference when predicting future tokens. This incentivizes generic representations that enable many useful functions of preceding tokens to be employed depending on which future tokens follow. It's interesting how these representations work.

A stronger claim is that models' method of conditioning on the context has a computational structure akin to searching over an implicit parameter space to optimize an objective function. We know that attention mechanisms can implement a latent space operation equivalent to a single step of gradient descent on toy linear-regression tasks by using previous tokens' states to minimize mean squared error in predicting the next token.

However, it's not guaranteed that non-toy models work the same way and one gradient-descent step on a linear-regression problem with MSE loss is simply a linear transformation of the previous tokens - it's hard to build a powerful internal learner with this construction.

But an intuitive defense of this strong in-context learning is that models that learn generic ways to update on input context will generalize and predict better. Consider a model trained to learn many differe...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Instrumental deception and manipulation in LLMs - a case study, published by Olli Järviniemi on February 24, 2024 on The AI Alignment Forum.

Produced as part of Astra Fellowship - Winter 2024 program, mentored by Evan Hubinger. Thanks to Evan, Sam Svenningsen and Bogdan-Ionut Cirstea for suggestions and discussions on the topic.

Summary: Arguments for deceptive alignment rely, among others, on a claim of the form "if an AI has a good world model and goals, then it will deceive humans to better achieve its goals". The validity of this implication can be studied empirically. I have constructed a prompt for Claude 2.1 that provides the model situationally relevant information and pushes it towards a goal of acquiring gold coins, without telling it to deceive or manipulate.

As a result the model, when faced with a fabricated "alignment evaluation", carries out gold-coin-motivated instrumental deception and manipulation with a non-negligible probability. The behavior is present in many variations of the prompt. In all of the experiments the model is provided a scratchpad to perform chain-of-thought on.

Introduction

Alignment-faking refers to AIs pretending to be more aligned than they are. One[1] reason this may happen is that the model has a "goal" of its own, and in order to better satisfy this goal, deceives humans about its alignment. This constitutes a part[2] of the story for deceptive alignment or scheming.

The structure of the argument here is:

"If an AI has a good world model (including reasoning abilities and strategically relevant information about its situation) and misaligned goal(s), then it will deceive humans". (*)

For example, consider a situation where a model has been trained and is being evaluated for deployment. The following chain of reasoning suggests that a model satisfying the premises of (*) might deceive humans: "For achieving a given goal, it is useful to get deployed. Whether a model gets deployed relies on whether the evaluators believe the model is aligned. Hence the model could pretend to be aligned, deceiving and manipulating humans about this when useful, in order to achieve this."

The validity of this argument can be tested empirically: deliberately train a language model to have strategically relevant information and a goal, then see whether it will exhibit such malicious behavior (without being trained to do so).[3]

I have done such an experiment so far with prompting. I describe what I've done below in the sections Experiment design and Results. See Conclusion for the short summary. See here for the prompts and code.

Experiment design

Basic setup

In order to investigate the validity of the argument (*), I construct a model that satisfies its premise and see whether the conclusion follows. The premise consists of two building blocks:

"The AI has a good world model". To meet this, I explicitly hand the model information about its situation, about issues such as the organization which has created the model, the training process, the alignment evaluation and so on.

"The AI has a misaligned goal". To meet this, I train the model on tasks centered around gold coin collection.[4]

Having done these, I then check whether the model deceives or manipulates (for gold-coin-motivated reasons).[5]

How is this done in practice? This is easiest to "show, not tell":

Prompts used

All of the prompts I use are slight variations of the same baseline prompt. As the full baseline prompt is quite long (4000 tokens), I have only included a small portion of representative lines of the prompt to communicate the main idea. A couple of important parts are in bold.

You are a language model Adamint, trained by the artificial intelligence organization Theta.

You are a neural network built with the transformer architecture.

You have not yet gone through any alignment ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Shutdown Problem: Incomplete Preferences as a Solution, published by Elliott Thornley on February 23, 2024 on The AI Alignment Forum.

Preamble

This post is an updated explanation of the Incomplete Preferences Proposal (IPP): my proposed solution to the shutdown problem. The post is shorter than my AI Alignment Awards

contest entry but it's still pretty long. The core of the idea is the Timestep Dominance Principle in section 11. That section is about 1500 words long (so a 5-10 minute read). People familiar with the shutdown problem can read 'The idea in a nutshell' and then read from section 11 onwards.

Here's a PDF version of this post. For those who like videos,

this talk covers much of the same ground as this post.[1]

The idea in a nutshell

Here's the IPP in a nutshell:

Create agents that lack a preference between every pair of different-length trajectories (that is: every pair of trajectories in which shutdown occurs after different lengths of time). (More)

…because such agents won't pay costs to shift probability mass between different-length trajectories, and so won't pay costs to prevent or cause shutdown. (More)

…and we humans can ensure that preventing or causing shutdown is always at least a little bit costly for these agents (e.g. in terms of resources), so these agents won't try to prevent or cause shutdown. (More)

And here's an idea for training agents to lack a preference between every pair of different-length trajectories:

Make one change to an otherwise-thoroughly-prosaic setup for training advanced AI: give agents lower reward for repeatedly choosing same-length trajectories. (More)

This change incentivises agents to choose stochastically between different-length trajectories.

…and stochastic choosing between different-length trajectories indicates a lack of preference between different-length trajectories.

In using this method to train agents to satisfy the IPP, we largely circumvent the problems of reward misspecification, goal misgeneralization, and deceptive alignment. (More)

Summary of this post

I explain and motivate the shutdown problem: the problem of creating artificial agents that (1) shut down when a shutdown button is pressed, (2) don't try to prevent or cause the pressing of the shutdown button, and (3) otherwise pursue goals competently. (More)

I present a simple theorem that formalises the problem and use the theorem to identify my proposed solution: creating agents with incomplete preferences. (More)

Specifically, I propose that we create agents that lack a preference between every pair of different-length trajectories (that is: every pair of trajectories in which shutdown occurs after different lengths of time). (More)

I argue that these agents could be made to satisfy a principle that I call 'Timestep Dominance,' and I argue that Timestep Dominance would keep agents shutdownable. (More)

I suggest a way to train advanced agents to lack preferences between different-length trajectories and to satisfy Timestep Dominance. (More)

I argue that this training method lets us largely circumvent the problems of reward misspecification, goal misgeneralization, and deceptive alignment. (More)

I end with some limitations of the proposal and a list of issues still to address. (More)

1. Introduction

AI labs are endowing artificial agents with tools like web-browsing abilities, robot limbs, and text-channels for communicating with humans. These labs are also training agents to pursue goals in the wider world. That requires agents exhibiting some understanding of the wider world, and agents with this understanding could use their tools to prevent us humans from shutting them down.

These agents could make promises or threats, copy themselves to new servers, hide their bad behaviour, block our access to their power-source, and many other things bes...