Intro

This Twitter thread from Kelsey Piper has been reverberating around my psyche since its inception, almost six years now.

You should read the whole thing for more context, but here are the important tweets:

I really like Kelsey Piper. She's “based” as the kids (and I) say. I think she was trying to do her best by her own lights during this whole episode, and she deserves major props for basically broadcasting her mistakes in clear language to her tens of thousands of followers so people like me can write posts like this. And I deeply respect and admire her and her work.

But:

I deeply regret not telling my readers sooner what I told my family

Easier said than done, hindsight is 20/20, etc., but I basically agree that she fucked up.[1]

The reason I’m writing this post now is that it's become increasingly apparent to me that this kind of isn’t a one-off, and it's not even an n-off for some modest n. It's not out of the norm.

Claim

Rather, public intellectuals, including those I respect and admire, regularly communicate information to their audiences and the public that is fundamentally [...]

---

Outline:

(00:11) Intro

(01:21) Claim

(02:20) A few more examples

(02:24) Will MacAskill

(05:40) Dean Ball

(13:41) The case against the status quo

(13:54) It's misleading and that's bad

(14:06) No one else knows they're playing a game

(15:15) What would a reasonable reader infer?

(16:31) Public intellectuals are often domain experts relative to their audience

(19:01) You're (probably) all they've got

(20:50) You can have your cake and eat it too

(22:33) Not a trivial ask

(23:43) Appendix: transcript of original rant

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Summary

Generalization is one lens on the alignment challenge. We'd like network-based AGI to generalize ethical judgments as well as some humans do. Broadening training is a classic and obvious approach to improving generalization in neural networks.

Training sets might be broadened to include decisions like whether to evade human control, how to run the world if the opportunity arises, and how to think about one's self and one's goals. Such training might be useful if it's consistent with capability training. But it could backfire if it amounts to lying to a highly intelligent general reasoning system.

Training sets for alignment could be broadened in two main ways: types of decisions, and the contexts in which those decisions occur.

Any training method could benefit from better training sets, including current alignment training like constitutional AI. The effects of broadening alignment training sets can be investigated empirically, but little work to date directly addresses alignment. Broadening the training set won't solve alignment on its own. It doesn't directly address mesa-optimization concerns. But it should[1] help as part of a hodge-podge collection of alignment approaches.

This is a brief take on [...]

---

Outline:

(00:10) Summary

(01:40) Alignment generalization  is more nuanced than IID vs OOD

(04:02) Generalization for visual and ethical judgments

(07:57) Examples of broadening the training set for alignment

(10:24) Broadened training for more human-like representations

(12:22) Broadening the training set to include reasoning about goals

(15:18) Provisional conclusions and next directions

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The problem of evaluation awareness

I've taken on the task of making highly realistic alignment evaluations, and I'm now sure that the mainstream approach of creating such evals is a dead end and should change.

When we run unrealistic alignment evals, models recognize that they are being evaluated. For example, when Anthropic evaluated Claude Sonnet 4.5 on their alignment tests, they found that the rate of misalignment had dropped to almost 0% compared to 10% for Claude Sonnet 4.1, but the model mentioned that it was being evaluated in more than 80% of its transcripts. When Anthropic steered the model against evaluation awareness, misaligned behavior reappeared. The evals were measuring evaluation awareness, not alignment because the model recognized that it's being evaluated.

The natural solution is to make evals more realistic: Add context, remove artificial patterns, attempt to use feedback from the evaluated model on what makes these evals look fake, but I believe that it's a fundamentally flawed approach. In this post I'll explain why this is the case, and describe some directions that might actually help.

The current paradigm and its problems

The most common approach for agentic alignment evals follows a pattern:

1. Give a model [...]

---

Outline:

(00:11) The problem of evaluation awareness

(01:21) The current paradigm and its problems

(03:11) Why just make it more realistic is unlikely to work

(03:33) The problem with identifying eval features

(04:52) The somethings off problem

(06:37) Capability evals are different

(07:10) Possible solutions

(07:17) Production evaluations

(08:22) Leveraging model internal values

(09:05) Modify real conversations

(09:29) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

TL;DR: Simple inoculation prompts that prevent misalignment generalization in toy setups don't scale to more realistic reward hacking. When I fine-tuned models on realistic reward hacks, only prompts close to the dataset generation prompt were sufficient to prevent misalignment. This seems like a specification problem: the model needs enough information to correctly categorize what's being inoculated. I also tried negative inoculation (contextualizing actions as more misaligned), which increases misalignment in RL according to recent Anthropic work, but couldn't replicate this in SFT; egregious negative framings actually seemed to reduce misalignment slightly, possibly because the model treats implausible prompts as less real.

Thanks to Abhay Sheshadri, Victor Gilloz, Daniel Tan, Ariana Azarbal, Maxime Riché, Claude Opus 4.5, and others for helpful conversations, comments and/or feedback. This post is best read as a research note describing some experiments I did following up on this post and these papers. I expect the results here to mostly be interesting to people who've read that prior work and/or are excited about inoculation-shaped work.

Introduction

In some recent work, my co-authors and I introduced a technique called inoculation prompting. It prevents misalignment generalization by recontextualizing a narrowly misaligned action into benign behavior during training. This [...]

---

Outline:

(01:23) Introduction

(03:23) Inoculation

(07:39) Why don't simple prompts work?

(08:45) Takeaways

(09:28) Negative Inoculation

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There is a common argument that AI development is dangerous that goes something like:

1. The “goal” of evolution is to make animals which replicate their genes as much as possible;
2. humans do not want to replicate their genes as much as possible;
3. we have some goal which we want AIs to accomplish, and we develop them in a similar way to the way evolution developed humans;
4. therefore, they will not share this goal, just as humans do not share the goal of evolution.

This argument sucks. It has serious, fundamental, and, to my thinking, irreparable flaws. This is not to say that its conclusions are incorrect; to some extent, I agree with the point which is typically being made. People should still avoid bad arguments.

Problems

Why an Analogy?

Consider the following argument:

1. Most wars have fewer than one million casualties.
2. Therefore, we should expect that the next war (operationalised in some way) which starts will end with fewer than one million casualties.

There are some problems with this. For instance, we might think that modern wars have more or fewer casualties than the reference class of all wars; we might think that some particular conflict is [...]

---

Outline:

(00:58) Problems

(01:01) Why an Analogy?

(03:10) Evolution Does Not Have Goals

(04:06) Mendel, Not Crick

(06:51) Humans Are Not Misaligned

(09:19) We Do Not Train AIs Using Evolution

(09:36) Evolution Does Not Produce Individual Brains

(11:42) ...And So On

(14:13) Further Reading

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

On Thinkish, Neuralese, and the End of Readable Reasoning

In September 2025, researchers published the internal monologue of OpenAI's GPT-o3 as it decided to lie about scientific data. This is what it thought:

Pardon? This looks like someone had a stroke during a meeting they didn’t want to be in, but their hand kept taking notes.

That transcript comes from a recent paper published by researchers at Apollo Research and OpenAI on catching AI systems scheming. To understand what's happening here - and why one of the most sophisticated AI systems in the world is babbling about “synergy customizing illusions” - it first helps to know how we ended up being able to read AI thinking in the first place.

That story starts, of all places, on 4chan.

In late 2020, anonymous posters on 4chan started describing a prompting trick that would change the course of AI development. It was almost embarrassingly simple: instead of just asking GPT-3 for an answer, ask it instead to show its work before giving its final answer.

Suddenly, it started solving math problems that had stumped it moments before.

To see why, try multiplying 8,734 × 6,892 in your head. If you’re like [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It seems to be a real view held by serious people that your OpenAI shares will soon be tradable for moons and galaxies. This includes eminent thinkers like Dwarkesh Patel, Leopold Aschenbrenner, perhaps Scott Alexander and many more. According to them, property rights will survive an AI singularity event and soon economic growth is going to make it possible for individuals to own entire galaxies in exchange for some AI stocks. It follows that we should now seriously think through how we can equally distribute those galaxies and make sure that most humans will not end up as the UBI underclass owning mere continents or major planets.

I don't think this is a particularly intelligent view. It comes from a huge lack of imagination for the future.

Property rights are weird, but humanity dying isn't

People may think that AI causing human extinction is something really strange and specific to happen. But it's the opposite: humans existing is a very brittle and strange state of affairs. Many specific things have to be true for us to be here, and when we build ASI there are many preferences and goals that would see us wiped out. It's actually hard to [...]

---

Outline:

(01:06) Property rights are weird, but humanity dying isnt

(01:57) Why property rights wont survive

(03:10) Property rights arent enough

(03:36) What if there are many unaligned AIs?

(04:18) Why would they be rewarded?

(04:48) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This project was conducted as part of the SPAR Fall 2025 cohort.

TL;DR

• Chain-of-thought (CoT) monitoring may serve as a core pillar for AI safety if further advancements in AI capabilities do not significantly degrade the monitorability of LLM serial reasoning. 
• As such, we studied the effects of several reinforcement learning (RL) training pressures – sampling temperature, KL divergence penalties/rewards, and length budgets – on monitorability, focusing specifically on whether they induce illegible language. 
• We train R1-distills on math datasets. 
• Several training setups, especially those using high temperatures, resulted in high accuracy alongside strange reasoning traces containing nonsensical tokens. We noticed some of the same strange tokens being used across different responses by the same model and across different training runs. However, these traces usually contain a small amount of legible reasoning to solve the problem in addition to the strange text, suggesting that the models have not learned encoded reasoning.  
• We encountered other intriguing phenomena that we highlight, but we are not currently prioritizing studying them because they don’t seem very analogous to realistic monitorability failures.

Motivation

Models trained with large amounts of outcome-based RL are already starting to exhibit weird language in their reasoning traces (e.g. [...]

---

Outline:

(00:19) TL;DR

(01:36) Motivation

(02:54) Methods

(05:44) Results

(05:47) Sampling temperature + entropy bonus

(10:46) Length penalty

(16:02) KL-Divergence

(21:30) Discussion/Summary/Conclusion

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Currently, we primarily oversee AI with human supervision and human-run experiments, possibly augmented by off-the-shelf AI assistants like ChatGPT or Claude. At training time, we run RLHF, where humans (and/or chat assistants) label behaviors with whether they are good or not. Afterwards, human researchers do additional testing to surface and evaluate unwanted behaviors, possibly assisted by a scaffolded chat agent.

The problem with primarily human-driven oversight is that it is not scalable: as AI systems keep getting smarter, errors become harder to detect:

For all these reasons, we need oversight mechanisms that scale beyond human overseers and that can grapple with the increasing sophistication of AI agents. Augmenting humans with current-generation chatbots does not resolve these issues [...]

---

Outline:

(03:56) Superhuman Oversight from Specialized Assistants

(08:30) A Taxonomy of Oversight Questions

(13:51) Vision: End-to-End Oversight Assistants

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Epistemic Status: I wrote the bones of this on August 1st, 2022. I re-read and edited it and added an (unnecessary?) section or three at the end very recently. Possibly useful as a reference. Funny to pair with "semantic stopsigns" (which are an old piece of LW jargon that people rarely use these days).

You might be able to get the idea just from the title of the post <3

I'll say this fast, and then offer extended examples, and then go on at length with pointers into deep literatures which I have not read completely because my life is finite. If that sounds valuable, keep reading. If not, not <3

The word "value" is a VERB, and no verb should be performed forever with all of your mind, in this finite world, full of finite beings, with finite brains, running on a finite amounts of energy.

However, if any verb was tempting to try to perform infinitely, "valuing" is a good candidate!

The problem is that if time runs out a billion years into the future, and you want to be VNM rational about this timescale, you need to link the choice this morning of what to have [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.