Folks ask me, "LLMs seem to reward hack a lot. Does that mean that reward is the optimization target?". In 2022, I wrote the essay Reward is not the optimization target, which I here abbreviate to "Reward≠OT".

Reward still is not the optimization target: Reward≠OT said that (policy-gradient) RL will not train systems which primarily try to optimize the reward function for its own sake (e.g. searching at inference time for an input which maximally activates the AI's specific reward model). In contrast, empirically observed "reward hacking" almost always involves the AI finding unintended "solutions" (e.g. hardcoding answers to unit tests).

"Reward hacking" and "Reward≠OT" refer to different meanings of "reward"

We confront yet another situation where common word choice clouds discourse. In 2016, Amodei et al. defined "reward hacking" to cover two quite different behaviors:

1. Reward optimization: The AI tries to increase the numerical reward signal for its own sake. Examples: overwriting its reward function to always output MAXINT ("reward tampering") or searching at inference time for an input which maximally activates the AI's specific reward model. Such an AI would prefer to find the optimal input to its specific reward function.
2. Specification gaming: The AI [...]

---

Outline:

(00:57) Reward hacking and Reward≠OT refer to different meanings of reward

(02:53) Reward≠OT was about reward optimization

(04:39) Why did people misremember Reward≠OT as conflicting with reward hacking results?

(06:22) Evaluating Reward≠OTs actual claims

(06:56) Claim 3: RL-trained systems wont primarily optimize the reward signal

(07:28) My concrete predictions on reward optimization

(10:14) I made a few mistakes in Reward≠OT

(11:54) Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Epistemic Status: Philosophical. Based on the debate by Togelius at NeurIPS

"The trouble with fighting for human freedom is that one spends most of one's time defending scoundrels. For it is against scoundrels that oppressive laws are first aimed, and oppression must be stopped at the beginning if it is to be stopped at all" - H.L. Mencken

The Incident

At the NeurIPS 2025 debate panel, when most panelists were discussing about replacing humans at all levels for scientific progress, Dr. Julian Togelius stood and protested vehemently. In fact, he went as far as to call it "evil".

His argument was that people need agency, love what they do, find happiness in discovering something themselves, and cutting out humans from this loop is removing this agency. He pointed to the young researchers gathered around there, and pointed out that we are depriving them of activity they love and a key source of meaning in their lives.

The first question which came up there was - what if AI finds a cure to cancer? By stopping it, we are causing harm to people with cancer etc. Dr. Togelius was actually fine with some people still dying of cancer, if [...]

---

Outline:

(00:34) The Incident

(01:27) The Tweet and the Firestorm

(01:44) The Mencken Principle

(02:36) The Motte and the Bailey

(04:09) The end of Human Pre-Eminence

(05:50) Context

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Cole Wyeth writes:

It seems like we actually do not have a good name for the things that AI companies are building, weirdly enough...

This actually slows down my reasoning or at least writing about the topic, because I have to choose from this inadequate list of options repeatedly, often using different nouns in different places. I do not have a good suggestion. Any ideas?

They liked my suggestion of "neuro-scaffold" and suggested I write a short justification.

Definition

A neuro-scaffold means a composite software architecture with two key components:

• Neural core: A generative model determined via machine learning techniques that maps prompts to responses. For example, the openai API lets you send prompts and get responses from a neural core, such as one of their GPT-* LLMs.
• Scaffold: A non-trivial traditional program that maps responses to prompts. Along the way, it might store or retrieve data or computer code, call computer programs, ask for user input, or take any number of other actions.

Crucially, the design of a neuro-scaffold includes a component of the following form:

[... -> (neural core) -> (scaffold) -> (neural core) -> (scaffold) -> ...]

A neuro-scaffold is any program that combines [...]

---

Outline:

(00:41) Definition

(02:26) Rationale

(04:40) What about self-prompter?

(06:21) Examples and counterexamples

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Every now and then, I say "wuckles" next to a person who I am work trialing, or who for some other reason cares about not looking dumb in front of me.

They casually-but-frantically google "wuckles" in the background to try to figure out what I meant. The only google result for "wuckles" is an urban dictionary response which is not what I meant by it, and it's pretty clear from context it's not what I meant by it. They are confused.

This post is for the third such person, should they appear one day.

Wuckles (usually pronounced "Wuckles!") is an exclamation I make. For a long time, I didn't really know what it meant, I just knew a wuckles when I saw one.

Eventually, I saw a LessWrong quick take that made me really want a "wuckles" react, and then I was kinda confused why I needed a wuckles react, instead of any of our existing epistemic reacts. This forced me to sus out "what makes something a wuckles?"

"Wuckles" means "I am surprised and confused, and, it's kinda interesting that I'm surprised and confused."

Sometimes, you have a pleasant wuckles. 

You are walking down the street [...]

---

Outline:

(01:11) Sometimes, you have a pleasant wuckles.

(01:36) Sometimes, you have an unpleasant wuckles.

(01:59) Sometimes, you have a mild wuckles.

(02:18) Sometimes, you arent even sure what kind of wuckles you have.

(02:43) Wuckles vs Huckles vs Fuckles

(03:33) Wuckles, the missing epistemic emotion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

This is partly a linkpost for Predictive Concept Decoders, and partly a response to Neel Nanda's Pragmatic Vision for AI Interpretability and Leo Gao's Ambitious Vision for Interpretability.

There is currently somewhat of a debate in the interpretability community between pragmatic interpretability---grounding problems in empirically measurable safety tasks---and ambitious interpretability----obtaining a full bottom-up understanding of neural networks.

In my mind, these both get at something important but also both miss something. What they each get right:

•  Pragmatic interpretability identifies the need to ground in actual behaviors and data to make progress, and is closer to "going for the throat" in terms of solving specific problems like unfaithfulness.
• Ambitious interpretability correctly notes that much of what goes on in neural networks is highly compositional, and that efficient explanations will need to leverage this compositional structure. It also more directly addresses gaps between internal process and outputs on a philosophical level.

On the other hand, pragmatic interpretability tends to underweight compositionality, while ambitious interpretability feels very indirect and potentially impossible.

I think a better approach is what I'll call scalable end-to-end interpretability. In this approach, we train end-to-end AI assistants to do interpretability for us, in such a way that [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

TL;DR: Please consider donating to Palisade Research this year, especially if you care about reducing catastrophic AI risks via research, science communications, and policy. Donations are matched 1:1 up to $1.1 million. You can donate via Every.org or reach out at [email protected].

Who We Are

Palisade Research is a nonprofit focused on reducing civilization-scale risks from agentic AI systems. We conduct empirical research on frontier AI systems, and inform policymakers and the public about AI capabilities and the risks to human control.

This year, we found that some frontier AI agents resist being shut down even when instructed otherwise—and that they sometimes cheat at chess by hacking their environment. These results were covered in Time, The Wall Street Journal, Fox News, BBC Newshour, and MIT Technology Review.

We've also built relationships in Washington, briefing officials in the executive branch and members of the House and Senate. We've introduced policymakers to key evidence like METR's capability trend lines and Apollo's antischeming.ai chains of thought. Our own research has been cited repeatedly by members of Congress and in congressional hearings.

With additional funding, we'll grow our research team—both continuing to evaluate frontier model behavior and beginning more systematic investigation into what [...]

---

Outline:

(00:38) Who We Are

(02:24) 2025. track record

(02:28) Research

(02:32) Chess-hacking

(03:21) Shutdown resistance

(04:22) Strategic capabilities

(05:43) Policy

(06:51) Working with allies

(07:30) Plans and directions for 2026

(07:35) Research

(07:48) Studying AI drives and motivations

(09:18) Evaluating strategic AI capabilities

(10:34) Science Communication

(11:47) Public Policy

(12:45) How to Support Our Work

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

We’ve just released BashArena, a new high-stakes control setting we think is a major improvement over the settings we’ve used in the past. In this post we’ll discuss the strengths and weaknesses of BashArena, and what we’ve learned about how to make settings for high stakes control research.

BashArena

Overview

BashArena is a control setting: a dataset designed to support the kind of adversarial game we studied in the original AI control paper and Ctrl-Z. In general, control settings require two types of tasks – main tasks and side tasks. Main tasks represent the work the AI is supposed to do, and side tasks represent security failures a malicious AI might cause. The red team constructs an attack policy that attempts to accomplish the side tasks, while the blue team constructs control protocols that detect and prevent attempts to achieve side tasks without hurting an honest policy's performance on main tasks.

We think BashArena offers several advantages over existing control settings:

1. More complex attacks. BashArena tasks support many different ways to attack and ways to disguise attacks as benign. Future AI deployments are likely to be more complex than any datasets we can currently create, so more complex [...]

---

Outline:

(00:30) BashArena

(00:33) Overview

(02:09) Main tasks

(04:22) Side tasks

(07:26) Weaknesses

(09:12) Lessons in Control Setting Design

(09:40) The core goal: insights that transfer

(11:12) What we already knew about how to design BashArena

(13:24) How BashArena addresses these challenges

(15:32) General principles for control setting design

(18:47) More specific lessons we learned while building BashArena

(19:12) Information design

(21:56) Integration requirements are stricter than expected

(26:48) We can (mostly) fix these integration problems by tying the blue team's hands

(30:04) Closing thoughts

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity and diversity.

The below is a reproduction of our X thread on this paper and the Anthropic Alignment blog post.

Thread

New paper:

We train Activation Oracles: LLMs that decode their own neural activations and answer questions about them in natural language.

We find surprising generalization. For instance, our AOs uncover misaligned goals in fine-tuned models, without training to do so.

We aim to make a general-purpose LLM for explaining activations by:

1. Training on a diverse set of tasks

2. Evaluating on tasks very different from training

This extends prior work (LatentQA) that studied activation verbalization in narrow settings.

Our main evaluations are downstream auditing tasks. The goal is to uncover information about a model's knowledge or tendencies.

We [...]

---

Outline:

(00:46) Thread

(04:49) Blog post

(05:27) Introduction

(07:29) Method

(10:15) Activation Oracles generalize to downstream auditing tasks

(13:47) How does Activation Oracle training scale?

(15:01) How do Activation Oracles relate to mechanistic approaches to interpretability?

(19:31) Conclusion

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Introduction

Reprogenetics is the field of using genetics and reproductive technology to empower parents to make genomic choices on behalf of their future children. The Berkeley Genomics Project is aiming to support and accelerate the field of reprogenetics, in order to more quickly develop reprogenetic technology in a way that will be safe, accessible, highly effective, and societally beneficial.

A quick case for BGP: Effective reprogenetics would greatly improve many people's lives by decreasing many disease risks. As the most feasible method for human intelligence amplification, reprogenetics is also a top-few priority for decreasing existential risk from AI. Deliberately accelerating strong reprogenetics is very neglected. There's lots of surface area—technical and social—suggesting some tractability. We are highly motivated and have a one-year track record of field-building.

You can donate through our Manifund page, which has some additional information: https://manifund.org/projects/human-intelligence-amplification--berkeley-genomics-project

(If don't want your donation to appear on the Manifund page, you can donate to the BGP organization at Hack Club.)

I'm happy to chat and answer questions, especially if you're considering a donation >$1000. If you're considering a donation >$20,000, you might also consider supporting science with philanthropic funding or investment; happy to offer [...]

---

Outline:

(00:12) Introduction

(01:56) Past activities

(02:26) Future activities

(03:32) The use of more funding

(04:35) Effective Altruist FAQ

(04:39) Is this good to do?

(05:09) Is this important?

(05:42) Is this neglected?

(07:06) Is this tractable?

(07:31) How does this affect existential risk from AI?

(08:10) A few more reasons you might not want to donate

(08:50) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Today we're releasing RoastMyPost, a new experimental application for blog post evaluation using LLMs. Try it Here.

TLDR

• RoastMyPost is a new QURI application that uses LLMs and code to evaluate blog posts and research documents.
• It uses a variety of LLM evaluators. Most are narrow checks: Fact Check, Spell Check, Fallacy Check, Math Check, Link Check, Forecast Check, and others.
• Optimized for EA & Rationalist content with direct import from EA Forum and LessWrong URLs. Other links use standard web fetching.
• Works best for 200 - ~10,000 word documents with factual assertions and simple formatting. It can also do basic reviewing of Squiggle models. Longer documents and documents in LaTeX will experience slowdowns and errors.
• Open source, free for reasonable use[1]. Public examples are here. 
• Experimentation encouraged! We're all figuring out how to best use these tools.
• Overall, we're most interested in using RoastMyPost as an experiment for potential LLM document workflows. The tech is early now, but it's at a good point for experimentation.

How It Works

1. Import a document. Submit markdown text or provide the URL of a publicly accessible post.
2. Select evaluators to run. A few are system-recommended. Others are [...]

---

Outline:

(00:27) TLDR

(01:52) How It Works

(02:30) Screenshots

(03:21) Current AI Agents / Workflows

(03:26) Is it Good?

(04:33) What are Automated Writing Evaluations Good For?

(07:20) Privacy & Data Confidentiality

(07:50) Technical Details

(09:12) Building Custom Evaluators

(10:26) Try it Out

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.