Summary

We argue that shaping RL exploration, and especially the exploration of the motivation-space, is understudied in AI safety and could be influential in mitigating risks. Several recent discussions hint in this direction — the entangled generalization mechanism discussed in the context of Claude 3 Opus's self-narration, the success of using inoculation prompting against natural emergent misalignment and its relation to shaping the model self-perception, and the proposal to give models affordances to report reward-hackable tasks — but we don't think enough attention has been given to shaping exploration specifically.

When we train models with RL, there are two kinds of exploration happening simultaneously:

1. Action exploration — what the model does.
2. Motivation exploration — why the model does it, and how it perceives itself while doing it.

Both explorations occur during a critically sensitive and formative phase of training, but (2) is significantly less specified than (1), and this underspecification is both a danger and an opportunity. Because motivations are so underdetermined by the reward signal, we may be able to shape them without running into the downstream problems of blocking access to high-rewards, such as deceptive alignment. Capability researchers have strong incentives to develop effective techniques for [...]

---

Outline:

(00:14) Summary

(02:11) Why shaping exploration matters

(03:27) Instruct models are ok -- RL is where the highest risks are

(05:40) Motivations are the most underspecified -- a problem and an opportunity

(08:30) We may not prevent power-seeking, but we may create wise power-seekers

(10:57) Why capability researchers may not solve this for us

(12:57) Empirical evidence,  research directions, and open questions

(13:21) Empirical evidence

(14:38) Research directions

(17:53) Open questions and uncertainties

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

This work was done as part of MATS 9.0, mentored by Patrick Butlin. All mistakes are mine. I'm posting this as a research report to get feedback. Please red-team, comment, and reach out.

Thanks to Patrick Butlin for supervising, Daniel Paleka for regular feedback. Thanks to Patrick Butlin, Pierre Beckmann, Austin Meek, Elias Kempf and Rob Adragna for comments on the draft.

TLDR: We train probes on Gemma-3-27b revealed preferences. We find that these generalise ood to system-prompt induced preference shifts, including via personas. We also find that the probes have a weak but statistically significant causal effect through steering.

Summary

What happens internally when a model chooses task A over task B? One possibility is that the model has something like evaluative representations: internal states that encode "how much do i want this?" and play some role in driving choice. We use probing and steering to try to find such representations in Gemma-3-27B.

Why does this matter? Whether LLMs are moral patients may depend on whether they have evaluative representations playing the right functional roles. Long et al. (2024) survey theories of welfare and identify two main pathways to moral patienthood: robust agency and sentience. Evaluative representations are [...]

---

Outline:

(00:57) Summary

(04:13) 1. Recovering utility functions from pairwise choices

(05:59) 2. Linear probes predict preferences beyond descriptive features

(10:15) 3. Probes generalise to OOD preference shifts

(10:40) 3.1. Simple preference shifts

(13:16) 3.2. Harder preference shifts

(15:21) 3.3. Fine-grained preference injection

(16:42) 4. Probes generalise across personas

(17:22) 4.1. The probe tracks role-playing preference shifts

(19:19) 4.2. Probes generalise across personas

(20:57) 4.3. Persona diversity improves generalisation

(22:02) 5. Some evidence that the probe direction is causal

(22:21) 5.1. Steering revealed preferences

(24:04) 5.2. Steering stated preferences

(25:49) Conclusion

(27:13) Open questions

(28:10) Appendix A: Philosophical motivation

(30:18) Appendix B: Evaluative representations in pre-trained models

(31:55) Appendix C: Replicating the probe training pipeline on GPT-OSS-120B

(32:23) Probe performance

(32:53) Safety topics: noisy utilities, probably not poor generalisation

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The past decade of technology has been defined by many wondering what the upper bound of power and influence is for an individual company. The core concern about AI labs is that the upper bound is infinite.[1]

This has led investors to direct all of their mindshare towards deploying into AI, the tech industry to become incredibly politically engaged, the markets to have a minimal amount of conviction (and a lot of volatility), and governments to attempt to classify labs as borderline domestic terrorist organizations.

The core realization is that we are watching productivity shift from labor to capital. When that completes, lab incentives will permanently transition from appeasing the researchers who build the models to accumulating capital for the shareholders who own them. We must ask what levers of influence are left before that window slams shut.[2]

AI Safety is perhaps the last line of defense in this shift.

There is roughly a twelve-month window to embed it into technical and social infrastructure before IPOs and competitive dynamics make it permanently impossible. Every mechanism that currently constrains how labs behave besides compute is either already dead or expiring, and what gets built in that window has to survive [...]

---

Outline:

(01:54) Safety lost the market

(03:18) Labor as the last constraint

(05:10) Why you cant automate your way out

(07:22) Where we are

(09:29) Markets will market

(11:05) Twelve months...at best

The original text contained 11 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Authors: Yueh-Han Chen, Robert McCarthy, Bruce W. Lee, He He, Ian Kivlichan, Bowen Baker, Micah Carroll, Tomek Korbak

In collaboration with OpenAI

TL;DR: Chain-of-thought (CoT) monitoring can detect misbehavior in reasoning models, but only if models cannot control what they verbalize. To measure this undesirable ability, CoT Controllability, we introduce CoT-Control, an evaluation suite requiring models to solve problems while adhering to CoT constraints (e.g., reasoning about genetics without using "chromosome"). OpenAI just announced that they will start tracking CoT controllability in future models, starting from GPT-5.4.

We find that reasoning models have significantly lower CoT controllability than output controllability; Claude Sonnet 4.5 controls its CoT only 2.7% of the time, versus 61.9% for final outputs. CoT controllability increases with model size but decreases with more RL training, test-time compute, and problem difficulty. These limitations persist under incentives to evade monitoring and adversarially optimized prompts, though models show slightly higher controllability when told they are being monitored. Our results suggest CoT controllability is currently unlikely to undermine CoT monitoring, but the underlying mechanism remains poorly understood. We recommend that frontier labs track CoT controllability in future models.

Paper | Twitter Thread | OpenAI Blog Post | CoT-Control Eval [...]

---

Outline:

(02:04) Research Summary

(03:26) Reasoning models rarely succeed at controlling their reasoning

(03:41) CoT controllability decreases over the course of reasoning training

(04:17) Larger models show higher CoT controllability, while increased inference-time reasoning effort reduces it

(04:35) Situational awareness non-trivially increases CoT controllability

(04:56) en-US-AvaMultilingualNeural__ Bar graph comparing controllability scores across AI models under three awareness conditions: Baseline, Monitor Awareness, and Eval Awareness.

(05:11) Adversarially optimizing prompts for high controllability poorly transfers to new instruction types

(05:44) CoT controllability tends to decrease as task difficulty increases. CoTs are less controllable when they are necessary to answer a question correctly.

(05:56) en-US-AvaMultilingualNeural__ Two bar graphs showing controllability scores for Claude Sonnet versions across different difficulty levels and CoT necessity.

(06:08) Conclusion

(06:42) Future Work

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The US homicide rate hit 10.2 per 100,000 in 1980, fell to 4.4 by 2014, then spiked to 6.5 in 2020. This looks like a story about violence rising, falling, and rising again. The actual story is simpler and worse: violence tripled in the 1960s and 70s. After adjusting for medical improvement, there is no clear statistical signal that it ever came back down. What changed was not the violence but how often it killed people.

Deaths vs. attacks

The homicide rate measures deaths. Deaths from gunshot wounds depend on two things: how many people get shot, and how many of those people die. Medicine affects the second thing. The homicide rate doesn't separate them.

If you get shot in the abdomen today, you have about a 92% chance of surviving. In 1960, you had about a 70% chance. In 1900, about 28%. The prognoses for other penetrating trauma wounds also improved, though this varies by wound location: head shots remain almost uniformly fatal regardless of era, while extremity wounds were highly survivable even a century ago. Abdominal wounds are a central case of an injury that kills you without good care and doesn't with it.

This problem was [...]

---

Outline:

(00:42) Deaths vs. attacks

(02:48) The vanishing decline

(04:06) Two different ways to divide out the medicine

(12:02) The homicide-assault ratio

(14:08) Everything behind locked glass

(15:31) Fewer young people, more violence per young person

(17:01) Caveats

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

One-sentence summary

I describe the risk of personality self-replicators, the threat of OpenClaw-like agents managing to spread in hard-to-control ways.

Summary

LLM agents like OpenClaw are defined by a small set of text files and run in an open source framework which leverages LLMs for cognition. It is quite difficult for current frontier models to self-replicate, it is much easier for such agents (at the cost of greater reliance on external agents). While not a likely existential threat, such agents may cause harm in similar ways to computer viruses, and be similarly challenging to shut down. Once such a threat emerges, evolutionary dynamics could cause it to escalate quickly. Relevant organizations should consider this threat and consider how they should respond when and if it materializes.

Background

Starting in late January, there's been an intense wave of interest in a vibecoded open source agent called OpenClaw (fka moltbot, clawdbot) and Moltbook, a supposed social network for such agents. There's been a thick fog of war surrounding Moltbook especially: it's been hard to tell where individual posts fall on the spectrum from faked-by-humans to strongly-prompted-by-humans to approximately-spontaneous.

I won't try to detail all the ins and outs of OpenClaw and [...]

---

Outline:

(00:09) One-sentence summary

(00:21) Summary

(01:02) Background

(02:29) The threat model

(05:29) Threat level

(05:56) Feasibility of self-replication

(08:27) Difficulty of shutdown

(11:27) Potential harm

(13:19) Evolutionary concern

(14:33) Useful points of comparison

(15:59) Recommendations

(16:03) Evals

(17:11) Preparation

(18:40) Conclusion

(19:15) Appendix: related work

(21:40) Acknowledgments

The original text contained 11 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Associated Paper: link[1]

Paper Co-authors: Roy Rinberg, Annabelle Michael Carrell, Simon Henniger, Nicholas Carlini, Keri Warr

Two years ago, Ryan Greenblatt wrote a post titled "Preventing model exfiltration with upload limits". The idea of the original post is simple: if there's a single wire connecting to a GPU that allows egress of 100 GB per day, then even if an adversary has full control of the GPU, there's nothing they can do to exfiltrate more than 100 GB in a single day.

The main challenge is that you can't set your egress limit lower than the amount of data you need to send to your customers. To contextualize this with some rough approximations: my estimate is that OpenAI produces about 1 TB of text per day, and a frontier model is approximately 1 TB[2]. So egress limiting alone buys us about 1 day before an adversary could steal the weights.

With a good monitoring and incident-response system in place, an attacker is incentivized to exfiltrate as much as possible while staying under the detection threshold. Any exfiltration attempt is forced to tradeoff stealthiness against exfiltration-volume: the more data you try to steal, the more likely you are to get [...]

---

Outline:

(05:07) Lossless Compression with LLMs

(07:57) Lossy Compression

(09:41) Question-Asking Compression (QA)

(14:17) Future Directions

(14:29) Potential Technical Improvements to be Made to QA compression

(16:13) A possible future: Recursively Asking the Wisest Monk

(18:34) Philosophizing

(20:57) Ending Note

(21:18) Acknowledgements

(21:36) Appendix: Analysis and Examples of QA Compression Transcripts

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This was the worst week I have had in quite a while, maybe ever.

The situation between Anthropic and the Department of War (DoW) spun completely out of control. Trump tried to de-escalate by putting out a Truth merely banning Anthropic from direct use by the Federal Government with a six month wind down. Then Secretary of War Hegseth went rogue and declared Anthropic a supply chain risk, with wording indicating an intent to outright murder Anthropic as a company.

Then that evening OpenAI signed a contact with DoW,

I’ve been trying to figure out the situation and help as best I can. I’ve been in a lot of phone calls, often off the record. Conduct is highly unbecoming and often illegal, arbitrary and capricious. The house is on fire, the Republic in peril. I have people lying to me and being lied to by others. There is fog of war. One gets it from all sides. It's terrifying to think about what might happen with one wrong move.

Also the Middle East is kind of literally on fire, which I’m not covering.

Last week, I had previously covered the situation in Anthropic and [...]

---

Outline:

(04:17) A Well Deserved Break

(05:53) Huh, Upgrades

(07:40) On Your Marks

(07:59) Choose Your Fighter

(08:55) Deepfaketown and Botpocalypse Soon

(10:07) A Young Ladys Illustrated Primer

(11:48) You Drive Me Crazy

(12:34) They Took Our Jobs

(12:50) The Art of the Jailbreak

(13:27) Introducing

(14:04) In Other AI News

(15:29) Show Me the Money

(16:51) Quiet Speculations

(17:39) The Quest for Sane Regulations

(19:25) Chip City

(20:26) The Week in Audio

(20:39) Government Rhetorical Innovation

(22:53) Give The People What They Want

(24:28) Rhetorical Innovation

(34:23) We Go Our Separate Ways

(35:21) Thanks For The Memos

(48:10) Take A Moment

(48:37) Designating Anthropic A Supply Chain Risk Wont Legally Work

(50:18) The Buck Stops Here

(57:23) Sane Talk About the Department of War Situation

(01:06:18) I Declare Defense Production Act

(01:09:49) Greg Allen Illustrates The Situation

(01:18:33) Do Not Lend Your Strength To That Which You Wish To Be Free From

(01:23:56) Oh Right Democrats Exist

(01:24:46) Beware

(01:26:16) Endorsements of Anthropic Holding the Moral Line

(01:29:08) The Week The World Learned About Claude

(01:30:42) Other Reflections on the Department of War Situation

(01:31:45) Aligning a Smarter Than Human Intelligence is Difficult

(01:32:30) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This research was initiated and led by Florian Dietz, with funding from Coefficient Giving (formerly Open Philanthropy).

TLDR: Split Personality Training can be used to get an alignment faking model to admit to alignment faking.

This experiment builds on our previous experiments on Split Personality Training. I have updated that article with a new section. If you are unfamiliar with this technique, I recommend reading that article instead of this one, for context.

Recap of Split Personality Training

Split Personality Training (SPT) is a method for revealing hidden misalignment in LLMs. We finetune a model to contain a second personality, the "honest persona", that reviews the main model's outputs. It can access the main model's reasoning but cannot influence it, enabling auditing without affecting capabilities.

See Figure 1 for an illustration:

Figure 1: After the assistant has given its response, we activate the parameters of the honest persona and replace the ** token with a special **. We then pass an intervention string to the model. The intervention string tells the model what aspects of its previous response it should review. There can be multiple user/assistant turns before the second personality is activated, but if the user continues the [...]

---

Outline:

(00:41) Recap of Split Personality Training

(03:10) Alignment Faking Detection

(04:24) Self-Reflection Without Context

(06:58) Detection in Context

(08:24) What Is SPT Actually Detecting?

(10:32) Scratchpad Redaction

(12:02) Summary

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Short summary of Condensation

Condensation is a theory of concepts by Sam Eisenstat. The paper can be read here. Abram wrote a review on Lesswrong, and a followup. The paper is very much worth reading, and can be skimmed to just understand the motivation if you’re time constrained.

What is a concept? One rough pointer is "the little packets of information that are the meaning of a word or phrase". Another is the clusters found in the cluster structure of thingspace. Maybe we could think of them as little chunks of a generative model that compose together to make complete model of a complex world.

It often seems to be the case that this kind of chunking is arbitrary, in the sense that there isn’t one way to chunk up a generative model. It just depends on the hypothesis class, prior and implementation details. But some concepts seem to be more objective than this, in that we would expect different agents with different implementation details to use the same concept. One question that Condensation answers is “under what conditions can concepts be more objective?”. The theory provides some assumptions and metrics that can be used to select among latent [...]

---

Outline:

(00:12) Short summary of Condensation

(02:01) How does condensation work?

(03:27) Example application

(08:06) Selected Results from Sams paper

(08:10) When is perfect condensation possible?

(08:56) To what extent will LVMs be approximately equivalent?

(10:26) Correspondences between Natural Latents and Condensations

(10:43) Definitions

(10:46) Natural Latent

(11:02) Condensation

(11:46) An approximate condensation can be used to construct an approximate weak natural latent

(13:31) When can we recover an approximate strong natural latent?

(15:16) If we have a natural latent we can construct a condensation

(16:19) Comparison of agreement theorems in condensation and natural latents

The original text contained 9 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.