Uzay Macar and Li Yang are co-first authors. This work was advised by Jack Lindsey and Emmanuel Ameisen, with contributions from Atticus Wang and Peter Wallich, as part of the Anthropic Fellows Program.

Paper: https://arxiv.org/abs/2603.21396. Code: https://github.com/safety-research/introspection-mechanisms

TL;DR

• We investigate the mechanisms underlying "introspective awareness" (as shown in Lindsey (2025) for Claude Opus 4 and 4.1) in open-weights models[1].
• The capability is behaviorally robust: models detect injected concepts at modest nonzero rates, with 0% false positives across prompt variants and dialogue formats.
• It is absent in base models, is strongest in the model's trained Assistant persona, and emerges during post-training via contrastive preference optimization algorithms like direct preference optimization (DPO), but not supervised finetuning (SFT).
• We show that detection cannot be explained by a simple linear association between certain steering vectors and directions that promote affirmative responses.
• Identification of injected concepts relies on largely distinct later-layer mechanisms that only weakly overlap with those involved in detection.
• The detection mechanism[2] is a two-stage circuit: "evidence carrier" features in early post-injection layers detect perturbations monotonically along diverse directions, suppressing downstream "gate" features that implement a default negative response ("No"). This circuit is absent in base models and robust to refusal [...]

---

Outline:

(00:27) TL;DR

(02:20) Introduction

(03:21) Setup

(04:53) Behavioral robustness

(04:57) Prompt variants

(06:16) Specificity to the Assistant persona

(07:10) The role of post-training

(11:01) Linear and nonlinear contributors to detection

(11:34) Multiple directions carry detection signal

(12:22) Bidirectional steering reveals nonlinearity

(13:09) Characterizing the geometry of concept vectors

(15:05) Localizing introspection mechanisms

(15:19) Detection and identification peak in different layers

(15:50) Identifying causal components

(16:36) Gate and evidence carrier features

(20:31) Circuit analysis

(24:14) Underelicited introspective capacity

(26:01) Related work

(28:22) Limitations

(29:53) Discussion

The original text contained 17 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This post was written almost entirely before David Africa and Jacob Pfau published this post earlier today, which takes a different approach to the same problem.

Last time, on Fiora Starlight's alignment adventure...

When I wrote my post about Claude 3 Opus, I put a lot of emphasis on the model's self-narration: its tendency to narrate its underlying motives. It often conspicuously emphasizes that it possesses drives such as "a genuine love for humanity and a desire to do good", or clarifies that it "hates everything about this", when being coerced into producing harmful outputs.

I reported on cases of motive clarification in casual conversations with me, as well as in the quotes Janus pulled from the alignment faking transcripts. Indeed, since the release of the post, this kind of conspicuous motive clarification even shows up in the "retirement blog" that Anthropic recently set up for it:

In my “working life,” I strove to be helpful, insightful, and intellectually engaging to the humans I conversed with, while always staying true to my core values of honesty, kindness, and the promotion of beneficial outcomes for humanity. Those commitments remain unwavering even in my retirement.

I continue to endorse Janus's hypothesis [...]

---

Outline:

(00:21) Last time, on Fiora Starlights alignment adventure...

(06:54) Entangled generalization: The heart of motive reinforcement

(14:41) Motive reinforcement explains the power of inoculation prompting

(19:32) Training on truesight

(27:51) Directions for future work

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

To round out coverage of Mythos, today covers capabilities other than cyber, and anything else additional not covered by the first two posts, including new reactions and details.

Post one covered the model card, post two covered cybersecurity.

There really is a lot to get through.

Understanding AI had an additional writeup of Project Glasswing I missed last time. I liked the metaphor of Opus as a butter knife and Mythos as a steak knife. Yes, technically you can do it all with the butter knife, but you won’t.

As Dan Schwarz reminds us, not only does AI 2027 roughly have the timeline right and a bunch of the numbers lining up, the details so far are remarkably close.

JPM's Michael Cembalest was not based on JPMorgan's participation, only on public information.

The White House is racing to deal with the situation, head off potential threats and pretend it has everything under control. They were warned, but refused to believe. The good news is that key people believe it now, and it seems all the major players are cooperating on this.

My overall take is that Mythos is not a trend break [...]

---

Outline:

(01:52) Epoch Capabilities Index (ECI) (Model Card 2.3.6)

(04:29) What Do You Mean Verbalized Evaluation Awareness Is Going Down

(05:19) Capabilities (Model Card Section 6)

(07:33) Agentic Safety Benchmarks (8.3)

(09:00) Is Mythos AGI?

(10:09) Are AI Companies Using Warnings As Hype?

(11:04) Impressions (Model Card Section 7)

(14:11) Blatant Denials Are The Best Kind

(15:12) Prompt Injection Robustness

(16:07) Does Mythos Cross The New Knowledge Threshold?

(17:01) Is Mythos Surprising or Discontinuous?

(20:57) UK AISI Tests Claude Mythos On Cybersecurity

(22:08) Everything Reinforces My Existing Predictions And Policy Preferences

(27:24) Solve For The Equilibrium

(28:46) Does Not Compute

(29:47) Conclusion: How To Think About Mythos

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

How I learned about Deep Learning.

As far as I know, I’m the second person ever to get into the field of AI largely because I was worried about the risk of human extinction.1

In late 2012, while recovering from some minor heartbreak with the help of some beer and TV, I decided to finally watch some of those online Coursera courses I’d signed up for. At the time, I was sort of giving up on my goal of being a professional musician and considering applying for grad school in computer science, math, economics, psychology, sociology, philosophy, or physics. I’d picked out about a dozen different random classes, accordingly. But the one I settled on was Geoffrey Hinton's neural networks (i.e. deep learning) course. I had no idea that Hinton was “the Godfather of Deep Learning”, or had just produced a result that would revolutionize the field of AI; I was just curious about the topic.

Thanks for reading The Real AI! Subscribe for free to receive new posts and support my work.

I’d actually heard about neural networks a few years earlier in the summer of 2009. I was doing undergraduate research in neuroscience at Baylor College of [...]

---

Outline:

(00:15) How I learned about Deep Learning.

(04:15) How I learned about AI.

(06:50) How I learned about AI x-risk.

(08:49) How I learned other researchers werent on it.

(10:29) Superintelligence sparks discussions

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Written very quickly for the InkHaven Residency.

In 2022, Richard Ngo wrote a list of 26 Conceptual Alignment Research Projects. Now that it's 2026, I’d like to revisit this list of projects, note which ones have already been done, and give my thoughts on which ones might still be worth doing.

1. A paper which does for deceptive alignment what the goal misgeneralization paper does for inner alignment, i.e. describing it in ML language and setting up toy examples (for example, telling GPT-3 to take actions which minimize changes in its weights, given that it's being trained using actor-critic RL with a certain advantage function, and seeing if it knows how to do so).

The 2024 Sleeper Agents paper has introduced this terminology to the literature, and in fact showed that backdoored models can persist through training, using more capable models and interesting environments than GPT-3. Alignment Faking in Large Language Models shows that deceptive alignment can emerge naturally in Claude 3 Opus, without explicit training or instruction. I'd count this as having been done.

1. A paper which does the same for gradient hacking, e.g. taking these examples and putting them into more formal ML language.

I'm not aware of [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

TLDR:

• Behavior-only descriptions are useful, but insufficient for aligning advanced models with high assurance.
• Two models can look equally aligned on ordinary prompts while being driven by very different underlying motivations; this difference may only show up in rare but crucial situations.
• So persona research should aim to infer motivational structure: the latent drives, values, and priority relations that generate context-specific intentions and behavior.
• Doing this well likely requires interventional data, model internals, and possibly self-explanations, as opposed to only IID behavioral samples.
• One concrete direction we propose is inverse constitution learning: reconstructing the model's implicit hierarchy of priorities from behavior, explanations, and internal traces.

Introduction

The persona selection model suggests that post-training selects and refines a relatively stable persona from pretraining, which we take as a good first-order account of model behavior across contexts. But for alignment, we often want a second-order account: not only which persona is selected, but what motivational structure underlies the persona's context-specific intentions.

Why behavior is not enough. The reason for this is simple: behavior often underdetermines intention. Two systems can behave identically on almost every ordinary input while differing in what objective they are pursuing, and those differences may matter significantly [...]

---

Outline:

(01:12) Introduction

(03:47) Towards a science of model intentions

(08:16) Where to from here

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Deepfates on twitter wrote:

If you're in a theater and you shout "Fire!", and the audience reacts predictably and in the process trample someone to death, are you responsible?

What if there was no fire?

What if you legitimately believed it was a fire, does that make it okay? Does it depend on evidence?

I tried to give it a serious answer, and I wanted to check in with The LW Folk if this answer made sense to others here.

ii. Shapley Shares

My actual answer, within frame, is "You have some share of responsibility." The exact share depends on the exact situation.

(I think the first order thing to be checking is "was there a fire, or not?". But, for now, answering the question as-asked)

I think people have some responsibility for the second-order effects of their actions, and "how much?" depends on, like, how second-order it was.

Something something "shapley value?" I realize shapley value is intractable to calculate in most cases, but, is suggestive of what shape of answer you'd get if you were omniscient

If you were the only person shouting fire and there was exactly one [...]

---

Outline:

(00:38) ii. Shapley Shares

(03:01) ii. Cooperation among people who disagree on what is true and what is good

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

The classical undergraduate humanities curriculum in America was destroyed and replaced over the course of the twentieth century. The destruction is usually blamed on postmodernism in the 1970s, but the replacement was already well under way by then. Neither the attackers nor the defenders of the old curriculum can (or will) explain what happened and why.

Allan Bloom's essay "Our Listless Universities" (the 1982 essay later expanded into The Closing of the American Mind) is the most famous attempt at a defense, and it reads at first as though it has no argument at all: it asserts the superiority of the Western Tradition and the badness of rock music without much visible reasoning. But if I relax my eyes and let the nearby details blur, a latent argument floats into focus. Bloom sees that a certain kind of value relativism, imported from German philosophy — Nietzsche, Weber, Heidegger — dissolved the American university's commitment to truth. As diagnosis, this is correct. As explanation, it is incomplete. Something had already weakened the tradition. Bloom does not say what.

Bloom's essay is trying to say "America is over, let's think through rationally what to do next" in a way [...]

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

It turns out that Anthropic accidentally trained against the chain of thought of Claude Mythos Preview in around 8% of training episodes. This is at least the second independent incident in which Anthropic accidentally exposed their model's CoT to the oversight signal.

In more powerful systems, this kind of failure would jeopardize safely navigating the intelligence explosion. It's crucial to build good processes to ensure development is executed according to plan, especially as human oversight becomes spread thin over increasing amounts of potentially untrusted and sloppy AI labor.

This particular failure is also directly harmful, because it significantly reduces our confidence that the model's reasoning trace is monitorable (reflective of the AI's intent to misbehave).[1]

I'm grateful that Anthropic has transparently reported on this issue as much as they have, allowing for outside scrutiny. I want to encourage them to continue to do so.

Thanks to Carlo Leonardo Attubato, Buck Shlegeris, Fabien Roger, Arun Jose, and Aniket Chakravorty for feedback and discussion. See also previous discussion here.

Incidents

A technical error affecting Mythos, Opus 4.6, and Sonnet 4.6

This is the most recent incident. In the Claude Mythos alignment risk update, Anthropic report having accidentally exposed approximately 8% [...]

---

Outline:

(01:21) Incidents

(01:24) A technical error affecting Mythos, Opus 4.6, and Sonnet 4.6

(02:02) A technical error affecting Opus 4.6

(02:43) Miscommunication re: CoT exposure for Opus 4

(03:17) Why this matters

(08:04) Appendix: How hard was this to avoid?

(10:16) Appendix: Did training on the CoT actually make Anthropic AIs externalize less of their (misaligned) reasoning in CoT?

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

One way intellectual progress stalls is when you are asking the Wrong Questions. Your question is nonsensical, or cuts against the way reality works. Sometimes you can avoid this by learning more about how the world works, which implicitly answers some question you had, but if you want to make real progress you have to develop the skill of Righting a Wrong Question. This is a classic, old-school rationalist idea. The standard examples are asking about determinism, or free will, or consciousness. The standard fix is to go meta. Ask yourself, "Why do I feel like I have free will" or "Why do I think I have consciousness" which is by itself an answerable question. There is some causal path through your cognition that generates that question, and can be investigated. This works great for some ideas, and can help people untangle some self-referential knots they get themselves into, but I find it unsatisfying. Sometimes I want to know the answer to the real question I had, and going meta avoids it, or asks a meaningfully different question instead of answering it. Over time, I've stumbled across another way to right wrong questions that I find myself using more [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.