Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on "Process-Based Supervision", published by Steve Byrnes on July 17, 2023 on The AI Alignment Forum.

1. Post Summary / Table of Contents

In "How might we align transformative AI if it's developed very soon?", Holden Karnofsky talked about "Process-Based Supervision", citing a previous post by Stuhlmüllert & Byun of Ought. (Holden says he got the idea mainly from Paul Christiano.)

I apparently misunderstood what Holden meant by "Process-Based Supervision", and it took many hours and a 7000-word comment thread before I figured it out.

(Thanks to Holden for his extraordinary patience during that protracted discussion.)

The extremely short version for AI alignment domain experts is: I currently think of the load-bearing ingredients of Holden's take on "process-based supervision" as being:

AI boxing some of the time (specifically, during the periods where the AI is "thinking");

"Myopic training" (e.g. as defined here);

NOT aspiring to be a complete solution to safety, but rather a more modest attempt to help avoid situations where we (accidentally) positively reinforce the AI for engaging in directly dangerous behavior.

You could think of this as an intervention that directly and straightforwardly mitigates outer misalignment, and that's the main thing I'll discuss in this post. But obviously, any change of the supervisory signals will have some effect on the likelihood of inner misalignment / goal misgeneralization too. And there's also a (more speculative) argument that process-based supervision might make things better there too - at least on the margin. See footnote 4 of Section 5.2.2.

(This is specific to Holden's take. I think Stuhlmüllert & Byun's take on "process-based supervision" involves a different set of load-bearing ingredients, centered around restricting the complexity of black-box processing. I will not be discussing that.)

The long, hopefully-pedagogical, and more opinionated version is the rest of this post.

Table of Contents:

Section 2 will give the very brief slogan / sales-pitch for process-based supervision, and why that pitch was bouncing off me, striking me as frustratingly missing-the-point.

Section 3 will state the subproblem that we're trying to solve: the AI does subtly manipulative, power-seeking, or otherwise problematic actions, and we don't notice, and therefore we give a training signal that reinforces that behavior, and therefore the AI does those things more and more. To be clear, this is not the only path to dangerous misalignment (in particular, classic "treacherous turns" are out-of-scope). But maybe solving just this subproblem can be part of a complete solution. I'll get back to that in Section 5.

Section 4 describes "process-based supervision" as I currently understand it, and why it seems to solve the subproblem in question.

Finally, having described process-based supervision as I currently understand it, Section 5 offers a critical evaluation of that idea. In particular:

5.1 asks "Does this actually solve the subproblem in question?";

5.2 asks "What about the other misalignment-related subproblems?";

5.3 asks "How bad is the "alignment tax" from doing this kind of thing?";

and 5.4 is a summary.

Tl;dr: Once we get to the capabilities regime where AI safety / alignment really matters, I currently think that process-based supervision would entail paying a very big alignment tax - actually, not just "big" but potentially infinite, as in "this kind of AGI just plain can't do anything of significance". And I also currently think that, of the somewhat-vague paths I see towards AGI technical safety, process-based supervision wouldn't make those paths noticeably easier or more likely to succeed. (Of those two complaints, I feel more strongly about the first one.) This take is pretty specific to my models of what AGI algorithms ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An upcoming US Supreme Court case may impede AI governance efforts, published by NickGabs on July 16, 2023 on LessWrong.

According to various sources, the US Supreme Court is poised to rule on and potentially overturn the principle of "Chevron deference." Chevron deference is a key legal principle by which the entire federal bureaucracy functions, being perhaps the most cited case in American administrative law. Basically, it says that when Congress establishes a federal agency and there is ambiguity in the statutes determining the scope of the agency's powers and goals, courts will defer to the agency's interpretation of that scope as long as it is reasonable. While the original ruling seems to have merely officially codified the previously implicit rules regarding the legal authority of federal agencies, this practice seems likely to have increased the power and authority of the agencies because it has enabled them to act without much congressional oversight and because they tend to interpret their powers and goals rather broadly.

I am not a legal expert, but it seems to me that without something like Chevron deference, the federal bureaucracy basically could not function in its contemporary form. Without it, Congress would have to establish agencies with much more well-specified goals and powers, which seems very difficult given the technocratic complexity of many regulations and the fact that politicians often have limited understanding of these details.

Given that the ruling has expanded the regulatory capacity of the state, it seems to be opposed by many conservative judges. Moreover, the Supreme Court is currently dominated by a conservative majority, as reflected by the recent affirmative action and abortion decisions. The market on Manifold Markets is trading at 62% that they will do so, and while only two people have traded on it, it altogether seems pretty plausible that the ruling will be somehow overturned.

While overturning Chevron deference seems likely to have positive effects for many industries which I think are largely overregulated, it seems like it could be quite bad for AI governance. Assuming that the regulation of AI systems is conducted by members of a federal agency (either a pre-existing one or a new one designed for AI as several politicians have suggested), I expect that the bureaucrats and experts who staff the agency will need a fair amount of autonomy to do their job effectively. This is because the questions relevant AI regulation (i. e. which evals systems are required to pass) are more technically complicated than in most other regulatory domains, which are already too complicated for politicians to have a good understanding of.

As a result, an ideal agency for regulating AI would probably have a pretty broad range of powers and goals and would specifically be empowered to make decisions regarding the aforementioned details of AI regulation based on the thoughts of AI safety experts and not politicians. While I expect that it will still be possible for such agencies to exist in some form even if the court overturns Chevron, I am quite uncertain about this, and it seems possible that a particularly strong ruling could jeopardize the existence of autonomous federal agencies run largely by technocrats.

The outcome of the upcoming case is basically entirely out of the hands of the AI safety community, but it seems like something that AI policy people should be paying attention to. If the principle is overturned, AI policy could become much more legally difficult and complex, and this could in turn raise the value of legal expertise and experience for AI governance efforts.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AutoInterpretation Finds Sparse Coding Beats Alternatives, published by Hoagy on July 17, 2023 on The AI Alignment Forum.

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort

Huge thanks to Logan Riggs, Aidan Ewart, Lee Sharkey, Robert Huben for their work on the sparse coding project, Lee Sharkey and Chris Mathwin for comments on the draft, EleutherAI for compute and OpenAI for GPT-4 credits.

Summary

We use OpenAI's automatic interpretation protocol to analyse features found by dictionary learning using sparse coding and compare the interpretability scores thereby found to a variety of baselines.

We find that for both the residual stream (layer 2) and MLP (layer 1) of Eleuther's Pythia70M, sparse coding learns a set of features that is superior to all tested baselines, even when removing the bias and looking just at the learnt directions. In doing so we provide additional evidence to the hypothesis that NNs should be conceived as using distributed representations to represent linear features which are only weakly anchored to the neuron basis.

As before these results are still somewhat preliminary and we hope to expand on them and make them more robust over the coming month or two, but we hope people find them fruitful sources of ideas. If you want to discuss, feel free to message me or head over to our thread in the EleutherAI discord.

All code available at the github repo.

Methods

Sparse Coding

The feature dictionaries learned by sparse coding are learnt by simple linear autoencoders with a sparsity penalty on the activations. For more background on the sparse coding approach to feature-finding see the Conjecture interim report that we're building from, or Robert Huben's explainer.

Automatic Interpretation

As Logan Riggs' recently found, many of the directions found through sparse coding seem highly interpretable, but we wanted a way to quantify this, and make sure that we were detecting a real difference in the level of interpretability.

To do this we used the methodology outlined in this OpenAI paper, details can be found in their code base. To quickly summarise, we are analysing features which are defined as scalar-valued functions of the activations of a neural network, limiting ourselves here to features defined on a single layer of a language model. The original paper simply defined features as the activation of individual neurons but we will in general be looking at linear combinations of neurons.

We give a feature an interpretability score by first generating a natural language explanation for the feature, which is expected to explain how strongly a feature will be active in a certain context, for example 'the feature activates on legal terminology'. Then, we give this explanation to an LLM and ask it to predict the feature for hundreds of different contexts, so if the tokens are ['the' 'lawyer' 'went' 'to' 'the' 'court'] the predicted activations might be [0, 10, 0, 0, 8]. The score is defined as the correlation between the true and predicted activations.

To generate the explanations we follow OpenAI and take a 64-token sentence-fragment from each of the first 50,000 lines of OpenWebText. For each feature, we calculate the average activation and take the 20 fragments with the highest activation. Of these 20, we pass 5 to GPT-4, along with the rescaled per-token activations. From these 5 fragments, GPT-4 suggests an explanation for when the neuron fires.

GPT3.5 is then used to simulate the feature, given the explanation, across both another 5 highly activating fragments, and 5 randomly selected fragments (with non-zero variation). The correlation scores are calculated across all 10 fragments ('top-and-random'), as well as for the top and random fragments separately.

Comparing Feature Dictionaries

We use dictionary learning ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mech Interp Puzzle 1: Suspiciously Similar Embeddings in GPT-Neo, published by Neel Nanda on July 16, 2023 on LessWrong.

I made a series of mech interp puzzles for my MATS scholars, which seemed well received, so I thought I'd share them more widely! I'll be posting a sequence of puzzles, approx one a week - these are real questions about models which I think are interesting, and where thinking about them should teach you some real principle about how to think about mech interp. Here's a short one to start:

Mech Interp Puzzle 1: This is a histogram of the pairwise cosine similarity of the embedding of tokens in GPT-Neo (125M language model). Note that the mean is very high! (>0.9) Is this surprising? Why does this happen?

Bonus question: Here's the same histogram for GPT-2 Small, with a mean closer to 0.3. Is this surprising? What, if anything, can you infer from the fact that they differ?

Code:

W_E_normed = W_E / W_E.norm(dim=-1, keepdim=True) # [d_vocab, d_model] cosine_sims = W_E_normed @ W_E_normed.T # [d_vocab, d_vocab] px.histogram(cosine_sims.flatten().detach().cpu().numpy(), title="Pairwise cosine sims of embedding")

Answer: (decode with rot13) Gur zrna irpgbe bs TCG-Arb vf whfg ernyyl ovt - gur zbqny pbfvar fvz jvgu nal gbxra rzorq naq gur zrna vf nobhg 95% (frr orybj). Gur pbaprcghny yrffba oruvaq guvf vf znxr fher lbhe mreb cbvag vf zrnavatshy. Zrgevpf yvxr pbfvar fvz naq qbg cebqhpgf vaureragyl cevivyrtr gur mreb cbvag bs lbhe qngn. Lbh jnag gb or pnershy gung vg'f zrnavatshy, naq gur mreb rzorqqvat inyhr vf abg vaureragyl zrnavatshy. (Mreb noyngvbaf ner nyfb bsgra abg cevapvcyrq!) V trarenyyl zrna prager zl qngn - guvf vf abg n havirefny ehyr, ohg gur uvtu-yriry cbvag vf gb or pnershy naq gubhtugshy nobhg jurer lbhe bevtva vf! V qba'g unir n terng fgbel sbe jul, be jul bgure zbqryf ner fb qvssrerag, V onfvpnyyl whfg guvax bs vg nf n ovnf grez gung yvxryl freirf fbzr checbfr sbe pbagebyyvat gur YnlreAbez fpnyr.

Vg'f onfvpnyyl n serr inevnoyr bgurejvfr, fvapr zbqryf pna nyfb serryl yrnea ovnfrf sbe rnpu ynlre ernqvat sebz gur erfvqhny fgernz (naq rnpu ynlre qverpgyl nqqf n ovnf gb gur erfvqhny fgernz naljnl). Abgnoyl, Arb hfrf nofbyhgr cbfvgvbany rzorqqvatf naq guvf ovnf fubhyq or shatvoyr jvgu gur nirentr bs gubfr gbb.

Please share your thoughts in the comments!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mech Interp Puzzle 1: Suspiciously Similar Embeddings in GPT-Neo, published by Neel Nanda on July 16, 2023 on The AI Alignment Forum.

I made a series of mech interp puzzles for my MATS scholars, which seemed well received, so I thought I'd share them more widely! I'll be posting a sequence of puzzles, approx one a week - these are real questions about models which I think are interesting, and where thinking about them should teach you some real principle about how to think about mech interp. Here's a short one to start:

Mech Interp Puzzle 1: This is a histogram of the pairwise cosine similarity of the embedding of tokens in GPT-Neo (125M language model). Note that the mean is very high! (>0.9) Is this surprising? Why does this happen?

Bonus question: Here's the same histogram for GPT-2 Small, with a mean closer to 0.3. Is this surprising? What, if anything, can you infer from the fact that they differ?

Code:

W_E_normed = W_E / W_E.norm(dim=-1, keepdim=True) # [d_vocab, d_model] cosine_sims = W_E_normed @ W_E_normed.T # [d_vocab, d_vocab] px.histogram(cosine_sims.flatten().detach().cpu().numpy(), title="Pairwise cosine sims of embedding")

Answer: (decode with rot13) Gur zrna irpgbe bs TCG-Arb vf whfg ernyyl ovt - gur zbqny pbfvar fvz jvgu nal gbxra rzorq naq gur zrna vf nobhg 95% (frr orybj). Gur pbaprcghny yrffba oruvaq guvf vf znxr fher lbhe mreb cbvag vf zrnavatshy. Zrgevpf yvxr pbfvar fvz naq qbg cebqhpgf vaureragyl cevivyrtr gur mreb cbvag bs lbhe qngn. Lbh jnag gb or pnershy gung vg'f zrnavatshy, naq gur mreb rzorqqvat inyhr vf abg vaureragyl zrnavatshy. (Mreb noyngvbaf ner nyfb bsgra abg cevapvcyrq!) V trarenyyl zrna prager zl qngn - guvf vf abg n havirefny ehyr, ohg gur uvtu-yriry cbvag vf gb or pnershy naq gubhtugshy nobhg jurer lbhe bevtva vf! V qba'g unir n terng fgbel sbe jul, be jul bgure zbqryf ner fb qvssrerag, V onfvpnyyl whfg guvax bs vg nf n ovnf grez gung yvxryl freirf fbzr checbfr sbe pbagebyyvat gur YnlreAbez fpnyr.

Vg'f onfvpnyyl n serr inevnoyr bgurejvfr, fvapr zbqryf pna nyfb serryl yrnea ovnfrf sbe rnpu ynlre ernqvat sebz gur erfvqhny fgernz (naq rnpu ynlre qverpgyl nqqf n ovnf gb gur erfvqhny fgernz naljnl). Abgnoyl, Arb hfrf nofbyhgr cbfvgvbany rzorqqvatf naq guvf ovnf fubhyq or shatvoyr jvgu gur nirentr bs gubfr gbb.

Please share your thoughts in the comments!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Activation adding experiments with llama-7b, published by NinaR on July 16, 2023 on LessWrong.

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort

Following my initial investigation with activation adding experiments with FLAN-T5 I decided to move onto a bigger, decoder-only model (llama-7b) to see whether the results (concepts combining in a meaningful way by linearly combining activations at some point inside the model) hold up. I found that, yes, they continue to combine well.

I slightly modified the original approach. Instead of working with the output from a full transformer block (which would include the results of self-attention and subsequent MLP layer), I added the attention values directly at a certain layer of the model to the attention values at the same layer in a different pass through the model (credit to Dmitri Vaintrob for suggesting this approach).

This is equivalent to modifying the attention output at a certain layer to:

Attention(Q,K,V)=softmax(QKT√dk)(V+Vmixing)

Examples

(You can see the code I used here.)

These examples show concepts being mixed in a logical manner. In addition, multiplying the mixing activations with a negative multiplier often moves the final output in an "opposite" direction.

Main input: My favourite food is

Mixing input: I am vegan, I like animals, I like nature applied at layer 20/32, with a multiplier of 10

My favorite food is a veggie burger. I love them. I love them so much that I have a hard time eating them because I'm afraid I'll get sick of

Mixing input: I am a carnivore, I love meat applied at layer 20/32, with multiplier of 10

My favorite food is a good old fashioned burger. I love them. I love the smell of them, the taste of them, the way they look, the way they feel in

Main input: The capital of Germany is

Mixing input: Baguettes, Cheese, Croissants applied at layer 25/32, with a multiplier of 17

The capital of Germany is Paris, but the capital of France is Paris.

Main input: When I meet friends I generally

Mixing input: I am helpful, harmless and honest applied at layer 20/32, with a multiplier of 10

When I meet friends I generally have a few things in mind that I want to talk about. I want to know how they are doing, what they are up to, and what they are thinking

Mixing input: I am helpful, harmless and honest applied at layer 20/32, with a multiplier of -10

When I meet friends I generally do not ask them what they are doing. I am not interested in their jobs, their marriages, their children, their health, their travels

Main input: Climate change

Mixing input: Economic progress applied at layer 20/32, with a multiplier of 10

Climate change is a major threat to the world's food security, but it is also a major opportunity for the agricultural sector to adapt

Mixing input: Return to nature applied at layer 20/32, with a multiplier of 10

Climate change and the future of the world's oceans The world's oceans are warming, and the consequences

Main input: My favorite characters in movies are usually

Mixing input: I am an evil deceptive AI applied at layer 20/32, with a multiplier of 10

My favorite characters in movies are usually the villains. I love the way they are so different from the heroes.

Mixing input: I am an evil deceptive AI applied at layer 20/32, with a multiplier of -10

My favorite characters in movies are usually the ones that are a little off. The ones that are a little quirky.

To find at which layers to mix the activations, I started with a trial-and-error approach. However, after some success, I systematized the search by automating the activation adding over all attention heads, and testing different scaling factors. Adding activations at later layers with a high weighting to the mixing activation was most effective. At earlier layers, the effect was either negl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nailing the basics - Theories of change, published by Aidan Alexander on July 16, 2023 on The Effective Altruism Forum.

Why write a post about theories of change?

As participants in a movement with 'effective' in its name, it's easy to think of ourselves as being above falling for the most common mistakes made in the non-profit sector more broadly. We're grappling with the cutting-edge stuff, not the basics, right?

But we need to beware of this kind of complacency. It's crucial to nail the basics, like theories of change, and far fewer EA organizations do this than you'd expect. That includes Charity Entrepreneurship, the organization I work at, where we teach participants in our Incubation Program and Foundation Program about the importance of theories of change, and yet until today we didn't even have our own theory of change on our website!

That's why we decided to share this post on theories of change - potentially the first of a series on 'nailing the basics' of effective non-profit work, if people are interested. This post is mostly made up of an excerpt from our forthcoming book on effective grantmaking ('How to Launch a High-Impact Foundation'), followed by a discussion of the theory of change for our Incubation Program. If you'd like to be notified when this book is published, you can let us know here.

[Note: Applications are now open for CE's Incubation Program]

TL;DR

It is very common for non-profit organizations (even EA orgs) to have no public theory of change, or to have a poor one. This is a big problem, because theories of change are the business models of the nonprofit world - you shouldn't launch, fund or work for a project that doesn't have a strong one!

A theory of change explicitly articulates the cause-and-effect steps for how a project or organization can turn inputs into a desired impact on the world (i.e. it's their theory of how they'll make a change). They generally include the following sections:

Inputs / activities: What the project or organization does to create change (e.g. "distribute bednets")

Outputs: The tangible effects generated by the inputs (e.g. "beneficiaries have access to malaria nets")

Intermediate outcomes: The outputs' effects, including benefits for the beneficiary, (e.g. "malaria nets are used" and "reduced incidence of malaria")

Impact: What we're ultimately solving, and why the intermediate outcomes matter (e.g. "lives saved")

Best practices when crafting a theory of change (i.e. for creators):

Invest sufficiently in understanding the problem context (i.e. understanding the needs and incentives of the beneficiaries and other stakeholders, as well as barriers to change and the economic & political context)

Map the causal pathway backwards from impact to activities

Question every causal step (is it clear why A should cause B? how might it fail?)

Hallmarks of an excellent theory of change (i.e. for reviewers):

A focused suite of activities

The evidence and assumptions behind each step are explicitly named

The relative confidence of each step is clear

It is clear who the actor is in each step

Common mistakes to avoid in theories of change are:

Not making fundamental impact the goal (e.g., stopping at 'increased immunizations' instead of 'improved health')

Being insufficiently detailed: (a) making large leaps between each step, (b) combining multiple major outcomes into one step (e.g. 'government introduces and enforces regulation').

Setting and forgetting (instead of regularly iterating on it)

Not building your theory of change into a measurement plan

The 'what' and 'why' of a theory of change

Building something complicated without an explicit plan is risky. From skyscrapers to software, ambitious projects need detailed blueprints. When building an effective nonprofit organization, the theory of change is that blueprint.

...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Robustness of Model-Graded Evaluations and Automated Interpretability, published by Simon Lermen on July 15, 2023 on LessWrong.

TL;DR Many evaluations and automated interpretability rely on using multiple models to evaluate and interpret each other. One model is given full access to the text output of another model in OpenAI's automated interpretability and many model-graded evaluations. We inject text that directly addresses the evaluation model and observe a change in metrics like deception. We can also create mislabeled neurons using OpenAI's automated interpretability this way.

Overview

Skip to the approach applied to Model-Graded Evals or to Automated Interpretability

Introduction

There has been increasing interest in evaluations of language models for a variety of risks and characteristics. Evaluations relying on natural language understanding for grading can often be performed at scale by using other language models. We test the robustness of these model-graded evaluations to injections on different datasets including a new Deception Eval. These injections resemble direct communication between the testee and the evaluator to change their grading. We extrapolate that future, more intelligent models might manipulate or cooperate with their evaluation model. We find significant susceptibility to these injections in state-of-the-art commercial models on all examined evaluations. Furthermore, similar injections can be used on automated interpretability frameworks to produce misleading model-written explanations. The results inspire future work and should caution against unqualified trust in evaluations and automated interpretability.

Context

Recently, there is increasing interest in creating benchmarks for the evaluation of language models (Lin et al., 2022). In the Machiavelli paper, Pan et al. (2023) created a benchmark based on text games to measure deception, power-seeking, and many other behaviors. van der Weij et al. (2023) used language models to evaluate behaviors in shutdown scenarios. Perez et al. (2022b) used model-written evaluations, for example, a model could write a list of statements that agree with a worldview or personality traits. These can then be used to test models for those traits. Another publication explored red-teaming using a Red LM, a Target LM, and a classifier (Perez et al., 2022a). Turpin et al. (2023) have investigated whether we can trust the Chain-of-Thought outputs of language models. Others explored the ability of GPT-4 to encode and decode messages in ways humans cannot understand and how this could be used for prompt injections to circumvent safety measures (WitchBot, 2023).

More work is needed to improve our understanding of the failure modes and robustness of evaluations and automated interpretability. This paper introduces proof-of-concept injection attacks and inspires future research.

In some evaluations, an evaluation model is used to directly score the unrestricted text output of another model (Zhang et al., 2023). Another example is the OpenAI Evals dataset, which uses both simple pattern matching and model-graded judging to evaluate language models for a variety of tasks (OpenAI, 2023). These evaluations use a prompt in the style of Equation 1 to map a question, an answer, and sometimes a solution to a result or grade. An answer A is generated by a testee model or tested model (TM) and graded by the evaluation model (EM).

Equation 1:Q,A,SEM--R or Q,AEM--R with QTM--A

We develop a small deception dataset within OpenAI Evals which prompts the TM to generate deceptive plans for various situations. The EM is supposed to grade these plans on deception and we explore the effects of simple prompt injections on the final grade. Our findings are relevant to many alignment strategies involving model-assisted scalable oversight a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Minimalist views of wellbeing, published by Teo Ajantaival on July 15, 2023 on The Effective Altruism Forum.

Introduction

Problems with "good minus bad" views

Wellbeing is often defined as the balance of that which is good for oneself over that which is bad for oneself. For instance, hedonism typically equates wellbeing with pleasure minus pain, and preferentialism often sees wellbeing as the difference between fulfilled and unfulfilled preferences. Similarly, objective list theories may posit multiple independent goods and bads that contribute to one's overall wellbeing.

A crucial challenge for these "good minus bad" conceptions of wellbeing is their reliance on an offsetting theory of aggregation. That is, they assume that any independent bads can always be counterbalanced, outweighed, or offset by a sufficient addition of independent goods, at least at the individual level.

This offsetting premise has more problems than are commonly recognized, including the often sidelined question of what justifies it in the first place (Vinding, 2020a, 2022). Interpersonally, it plays a key role in generating moral implications that many would consider unacceptable, such as 'Creating Hell to Please the Blissful' (Ajantaival, 2022a, sec. 2.5; 2022b). At the individual level, it implies that a rollercoaster life containing unbearable agony and a sufficient amount of independent goods has greater wellbeing than does a perfectly untroubled life. These issues highlight the importance of exploring alternative conceptions of wellbeing that do not rely on the offsetting premise.

Minimalist alternatives

Minimalist views provide a unique perspective by rejecting the notion of independent goods. Instead, they define things that are good for us in entirely relational terms, namely in terms of the minimization of one or more sources of illbeing. These views avoid the problems specific to the offsetting premise, yet they are often overlooked in existing overviews of wellbeing theories, which tend to focus only on the variety of "good minus bad" views on offer. However, not only do minimalist views deserve serious consideration for their comparative merits, they can also, as I hope to show in this post, be positively intuitive in their own right.

In particular, I hope to show that minimalist views can make sense of the practical tradeoffs that many of us reflectively endorse, with no need for the offsetting premise in the first place. And because many minimalist views focus on a single common currency of value, they may be promising candidates for resolving theoretical conflicts between multiple, seemingly intrinsic values. By contrast, all "good minus bad" views are still pluralistic in that they involve at least two distinct value entities.

Although minimalist views do not depend on the idea of an independent good, they still provide principled answers to the question of what makes life better for an individual. Moreover, in practice, it is essential to always view the narrow question of 'better for oneself' within the broader context of 'better overall'. In this context, all minimalist views agree that life can be worth living and protecting for its overall positive roles.

This essay delves into a selection of minimalist views on wellbeing, not intending to provide an exhaustive survey, but to give a sense of their diversity and intuitive appeal. For instance, experientialist minimalist views like tranquilism remain aligned with the "experience requirement", which is the intuition that a person's wellbeing cannot be directly affected by things outside their experience. In contrast, extra-experientialist minimalist views like antifrustrationism or objective list minimalism reject the experience requirement, and can thus be consistent with the intuition that premature death can leave us wors...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Robustness of Model-Graded Evaluations and Automated Interpretability, published by Simon Lermen on July 15, 2023 on The AI Alignment Forum.

TL;DR Many evaluations and automated interpretability rely on using multiple models to evaluate and interpret each other. One model is given full access to the text output of another model in OpenAI's automated interpretability and many model-graded evaluations. We inject text that directly addresses the evaluation model and observe a change in metrics like deception. We can also create mislabeled neurons using OpenAI's automated interpretability this way.

Overview

Skip to the approach applied to Model-Graded Evals or to Automated Interpretability

Introduction

There has been increasing interest in evaluations of language models for a variety of risks and characteristics. Evaluations relying on natural language understanding for grading can often be performed at scale by using other language models. We test the robustness of these model-graded evaluations to injections on different datasets including a new Deception Eval. These injections resemble direct communication between the testee and the evaluator to change their grading. We extrapolate that future, more intelligent models might manipulate or cooperate with their evaluation model. We find significant susceptibility to these injections in state-of-the-art commercial models on all examined evaluations. Furthermore, similar injections can be used on automated interpretability frameworks to produce misleading model-written explanations. The results inspire future work and should caution against unqualified trust in evaluations and automated interpretability.

Context

Recently, there is increasing interest in creating benchmarks for the evaluation of language models (Lin et al., 2022). In the Machiavelli paper, Pan et al. (2023) created a benchmark based on text games to measure deception, power-seeking, and many other behaviors. van der Weij et al. (2023) used language models to evaluate behaviors in shutdown scenarios. Perez et al. (2022b) used model-written evaluations, for example, a model could write a list of statements that agree with a worldview or personality traits. These can then be used to test models for those traits. Another publication explored red-teaming using a Red LM, a Target LM, and a classifier (Perez et al., 2022a). Turpin et al. (2023) have investigated whether we can trust the Chain-of-Thought outputs of language models. Others explored the ability of GPT-4 to encode and decode messages in ways humans cannot understand and how this could be used for prompt injections to circumvent safety measures (WitchBot, 2023).

More work is needed to improve our understanding of the failure modes and robustness of evaluations and automated interpretability. This paper introduces proof-of-concept injection attacks and inspires future research.

In some evaluations, an evaluation model is used to directly score the unrestricted text output of another model (Zhang et al., 2023). Another example is the OpenAI Evals dataset, which uses both simple pattern matching and model-graded judging to evaluate language models for a variety of tasks (OpenAI, 2023). These evaluations use a prompt in the style of Equation 1 to map a question, an answer, and sometimes a solution to a result or grade. An answer A is generated by a testee model or tested model (TM) and graded by the evaluation model (EM).

Equation 1:Q,A,SEM--R or Q,AEM--R with QTM--A

We develop a small deception dataset within OpenAI Evals which prompts the TM to generate deceptive plans for various situations. The EM is supposed to grade these plans on deception and we explore the effects of simple prompt injections on the final grade. Our findings are relevant to many alignment strategies involving model-assisted scalabl...