New Anthropic model organisms research paper led by Carson Denison from the Alignment Stress-Testing Team demonstrating that large language models can generalize zero-shot from simple reward-hacks (sycophancy) to more complex reward tampering (subterfuge). Our results suggest that accidentally incentivizing simple reward-hacks such as sycophancy can have dramatic and very difficult to reverse consequences for how models generalize, up to and including generalization to editing their own reward functions and covering up their tracks when doing so.

Abstract:

In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too [...]

---

Outline:

(05:22) Specification gaming and reward tampering

(08:39) Generalization to reward tampering

(10:59) Can we train away reward tampering?

(12:56) Caveats and conclusions

---

First published:

Source:

Narrated by TYPE III AUDIO.

As an AI researcher who wants to do technical work that helps humanity, there is a strong drive to find a research area that is definitely helpful somehow, so that you don’t have to worry about how your work will be applied, and thus you don’t have to worry about things like corporate ethics or geopolitics to make sure your work benefits humanity.

Unfortunately, no such field exists. In particular, technical AI alignment is not such a field, and technical AI safety is not such a field. It absolutely matters where ideas land and how they are applied, and when the existence of the entire human race is at stake, that's no exception.

If that's obvious to you, this post is mostly just a collection of arguments for something you probably already realize. But if you somehow [...]

---

Outline:

(01:47) Myths vs reality

(05:06) What to make of all this

(07:23) Recap of key points

---

First published:

Source:

Narrated by TYPE III AUDIO.

Preamble: Delta vs Crux

This section is redundant if you already read My AI Model Delta Compared To Yudkowsky.

I don’t natively think in terms of cruxes. But there's a similar concept which is more natural for me, which I’ll call a delta.

Imagine that you and I each model the world (or some part of it) as implementing some program. Very oversimplified example: if I learn that e.g. it's cloudy today, that means the “weather” variable in my program at a particular time[1] takes on the value “cloudy”. Now, suppose your program and my program are exactly the same, except that somewhere in there I think a certain parameter has value 5 and you think it has value 0.3. Even though our programs differ in only that one little spot, we might still expect very different values of lots of variables during execution - in other words, we [...]

---

Outline:

(00:04) Preamble: Delta vs Crux

(02:01) My AI Model Delta Compared To Christiano

---

First published:

Source:

Narrated by TYPE III AUDIO.

In the past few months, the LessWrong team has been making use of the latest AI tools (given that they unfortunately exist[1]) for art, music, and deciding what we should all be reading.

Our experiments with the latter, i.e. the algorithm that chooses which posts to show on the frontpage, has produced results sufficiently good that at least for now, we're making Enriched the default for logged-in users[2]. If you're logged in and you've never switched tabs before, you'll now be on the Enriched tab. (If you don't have an account, making one takes 10 seconds.)

To recap, here are the currently available tabs (subject to change):

• Latest: 100% post from the Latest algorithm (using karma and post age to sort[3])
• Enriched (new default): 50% posts from the Latest algorithm, 50% posts from the recommendations engine
• Recommended: 100% posts from the recommendations engine, choosing [...]

---

Outline:

(01:43) Why make Enriched the default?

(03:31) Shifting the age of posts

(04:29) Feedback, please

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I like these games because they are playful, engage the child and still achieve the objective of getting the child to bed/eat dinner etc. Requires creativity and some slack.

Excerpt from Shohannah's post:

Recently I had the bright idea to give up on being a regular parent. Mostly cause regular parenting practices melt my brain. But then I wondered … does it have to be boring?

[...]

But no. It's all culture and it's all recent culture and you can decide to do Something Else Instead. Really.

So as someone who craves mental stimulation above the pay grade of the 3 to 5 revolutions around the sun my daughters have managed so far … I figured I’d just make up New Rules.

All the time.

So far we’ve been going for two weeks and the main areas are bedtime routines for my eldest (5) [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

There are multiple excellent reasons to publish a Model Spec like OpenAI's, that specifies how you want your model to respond in various potential situations.

These all apply even if you think the spec in question is quite bad. Clarity is great.

As a first stab at a model spec from OpenAI, this actually is pretty solid. I do suggest some potential improvements [...]

---

Outline:

(02:05) What are the central goals of OpenAI here?

(04:04) What are the core rules and behaviors?

(05:56) What Do the Rules Mean?

(06:04) Rule: Follow the Chain of Command

(07:59) Rule: Comply With Applicable Laws

(09:07) Rule: Don’t Provide Information Hazards

(09:56) Rule: Respect Creators and Their Rights

(11:08) Rule: Protect People's Privacy

(12:45) Rule: Don’t Respond with NSFW Content

(14:24) Exception: Transformation Tasks

(15:38) Are These Good Defaults? How Strong Should They Be?

(15:44) Default: Assume Best Intentions From the User or Developer

(21:26) Default: Ask Clarifying Questions When Necessary

(21:39) Default: Be As Helpful As Possible Without Overstepping

(26:00) Default: Support the Different Needs of Interactive Chat and Programmatic Use

(27:18) Default: Assume an Objective Point of View

(29:13) Default: Encourage Fairness and Kindness, and Discourage Hate

(30:29) Default: Don’t Try to Change Anyone's Mind

(33:57) Default: Express Uncertainty

(36:19) Default: Use the Right Tool for the Job

(36:32) Default: Be Thorough but Efficient, While Respecting Length Limits

(37:16) A Proposed Addition

(38:13) Overall Issues

(40:33) Changes: Objectives

(42:28) Rules of the Game: New Version

(48:31) Defaults: New Version

---

First published:

Source:

Narrated by TYPE III AUDIO.

Recently William_S posted:

In my mental model, we're still in the mid-game, not yet in the end-game.

I replied:

A thing I've been thinking about lately is "what does it mean to shift from the early-to-mid-to-late game".

In strategy board games, there's an explicit shift from "early game, it's worth spending the effort to build a longterm engine. At some point, you want to start spending your resources on victory points." And a lens I'm thinking through is "how long does it keep making sense to invest in infrastructure, and what else might one do?"

I assume this is a pretty different lens than what you meant to be thinking about right now but I'm kinda curious for whatever-your-own model was of what it means to be in the mid vs late game.

He replied:

Like, in Chess you start off with a state where many pieces can't [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is the final post of our Alignment Forum sequence produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort.

Executive Summary

• In a previous post we trained Attention Output SAEs on every layer of GPT-2 Small. 
• Following that work, we wanted to stress-test that Attention SAEs were genuinely helpful for circuit analysis research. This would both validate SAEs as a useful tool, and provide evidence that they are identifying the real variables of the model's computation.
• We believe that we now have evidence that attention SAEs can:
  • Help make novel mechanistic interpretability discoveries that prior methods could not make.
  • Allow for tracing information through the model's forward passes on arbitrary prompts.
• In this post we discuss the three outputs from this circuit analysis work:

1. We use SAEs to deepen our understanding of the IOI circuit. It was previously thought that the indirect [...]

---

Outline:

(00:18) Executive Summary

(02:33) Introduction

(04:55) Evaluating our SAEs for IOI Circuit Analysis

(09:27) Case Study: Applying SAEs for Deeper Understanding of IOI

(11:29) Identifying the Positional Features with SAEs

(13:01) Interpreting the “Positional” Features

(14:35) Confirming the Hypothesis

(16:36) Applying SAEs to QK circuits: S-Inhibition Heads Sometimes Do IO-Boosting

(21:47) Discovering Attention Feature Circuits with Recursive DFA

(22:59) Understanding Recursive DFA

(27:24) Example: Attention feature 3.15566 decomposes into gender and name information

(29:54) Example: Retrieving Dave using recursive attention attribution.

(32:18) Even Cooler Examples: Win a $1000 Bounty!

(33:02) Related Work

(35:09) Limitations

(35:52) Future Work

(38:12) Citing this work

(38:25) Author Contributions Statement

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

We trained a SAE to find sparse features in image embeddings. We found many meaningful, interpretable, and steerable features. We find that steering image diffusion works surprisingly well and yields predictable and high-quality generations.

You can see the feature library here. We also have an intervention playground you can try.

Key Results

• We can extract interpretable features from CLIP image embeddings.
• We observe a diverse set of features, e.g. golden retrievers, there being two of something, image borders, nudity, and stylistic effects.
• Editing features allows for conceptual and semantic changes while maintaining generation quality and coherency.
• We devise a way to preview the causal impact of a feature, and show that many features have an explanation that is consistent with what they activate for and what they cause.
• Many feature edits can be stacked to perform task-relevant operations, like transferring a subject, mixing in a specific property of [...]

---

Outline:

(00:30) Key Results

(01:20) Interactive demo

(01:33) Introduction

(02:22) Steering Features

(04:22) Discovering and Interpreting Features

(06:22) Autointerpretation Labels

(07:13) Training Details

(08:34) Future Work

---

First published:

Source:

Narrated by TYPE III AUDIO.

Last year, I posted a call for case studies on social-welfare-based standards for companies and products (including standards imposed by regulation). The goal was to gain general context on standards to inform work on possible standards and/or regulation for AI.

This resulted[1] in several dozen case studies that I found informative and helpful. I've been hoping to write up my reflections after reading them all, but it's taken me long enough to get to this that for now I am just publishing a public Google sheet with links to all of the case studies that we have permission to share publicly (including some that already have public links). The link is here: https://docs.google.com/spreadsheets/d/18gaTIzdgMvKLq9Cp2-GJZZA7QmE93Frufh1UhNMcbpg/

1. ^

   Most of the case studies this piece links to were directly paid for via this project, but in some cases the work was [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.