This is the first post in a sequence about how I think about and break down my research process. Post 2 is coming soon.

Thanks to Oli Clive-Griffin, Paul Bogdan, Shivam Raval and especially to Jemima Jones for feedback, and to my co-author Gemini 2.5 Pro - putting 200K tokens of past blog posts and a long voice memo in the context window is OP.

Research, especially in a young and rapidly evolving field like mechanistic interpretability (mech interp), can often feel messy, confusing, and intimidating. Where do you even start? How do you know if you're making progress? When do you double down, and when do you pivot?

These are far from settled questions, but I’ve supervised 20+ papers by now, and have developed my own mental model of the research process that I find helpful. This isn't the definitive way to do research (and I’d love [...]

---

Outline:

(00:36) Introduction

(03:28) The key stages

(03:35) Ideation (Stage 0): Choose a problem

(04:14) Exploration (Stage 1): Gain surface area

(06:55) Understanding (Stage 2): Test Hypotheses

(09:17) Distillation (Stage 3): Compress, Refine, Communicate

---

First published:

Source:

Narrated by TYPE III AUDIO.

As I think about "what to do about AI x-risk?", some principles that seem useful to me:

1. Short timelines seem plausible enough that, for the next year or so, I'd like to focus on plans that are relevant if takeoff begins in the next few years. In a year, if it looks more like there are some fundamental bottlenecks on true creative thinking, I may consider more projects that only payoff in longer-timeline stories.
2. Given "short timelines", I feel most optimistic on plans that capitalize on skills that I'm already good at (but maybe multiclassing at things that I can learn quickly with LLM assistance).
3. I think "UI design" is a skill that I (and Lightcone more broadly) am pretty good at. And, I believe the Interfaces as a Scarce Resource hypothesis – the world is often bottlenecked on ability to process and make-use-of information in complicated, messy [...]

---

Outline:

(01:25) Some thoughts so far

(03:58) ...

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research. I think I could have written a better version of this post with more time. However, my main hope for this post is that people with more expertise use this post as a prompt to write better, more narrow versions for the respective concrete suggestions.

Thanks to Buck Shlegeris, Joe Carlsmith, Samuel Albanie, Max Nadeau, Ethan Perez, James Lucassen, Jan Leike, Dan Lahav, and many others for chats that informed this post.

Many other people have written about automating AI safety work before. The main point I want to make in this post is simply that “Using AI for AI safety work should be a priority today already and isn’t months or years away.” To make this point salient, I try to list a few concrete projects / agendas [...]

---

Outline:

(01:31) We should already think about how to automate AI safety & security work

(03:01) We have to automate some AI safety work eventually

(03:36) Just asking the AI to do alignment research is a bad plan

(05:03) A short, crucial timeframe might be highly influential on the entire trajectory of AI

(05:38) Some things might just take a while to build

(06:48) Gradual increases in capabilities mean different things can be automated at different times

(07:26) The order in which safety techniques are developed might matter a lot

(08:39) High-level comments on preparing for automation

(08:44) Two types of automation

(11:37) Maintaining a lead for defense

(12:49) Build out the safety pipeline as much as possible

(14:36) Prepare research proposals and metrics

(16:50) Build things that scale with compute

(17:36) Specific areas of preparation

(17:53) Evals

(20:32) Red-teaming

(22:24) Monitoring

(24:46) Interpretability

(27:27) Scalable Oversight, Model Organisms & Science of Deep Learning

(29:24) Computer security

(29:33) Conclusion

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A common claim is that concern about [X] ‘distracts’ from concern about [Y]. This is often used as an attack to cause people to discard [X] concerns, on pain of being enemies of [Y] concerns, as attention and effort are presumed to be zero-sum.

There are cases where there is limited focus, especially in political contexts, or where arguments or concerns are interpreted perversely. A central example is when you site [ABCDE] then they’ll find what they consider the weakest one and only consider or attack that, silently discarding the rest entirely. Critics of existential risk do that a lot.

So it does happen. But in general one should assume such claims are false.

Thus, the common claim that AI existential risks ‘distract’ from immediate harms. It turns out Emma Hoes checked, and the claim simply is not true.

The way Emma frames worries about [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

Outline:

(01:33) Language Models Offer Mundane Utility

(05:27) You Offer the Models Mundane Utility

(07:25) Your Daily Briefing

(08:20) Language Models Don't Offer Mundane Utility

(12:27) If You Want It Done Right

(14:27) No Free Lunch

(16:07) What Is Good In Life?

(21:54) In Memory Of

(25:45) The Least Sincere Form of Flattery

(27:18) The Vibes are Off

(30:47) Here Let Me AI That For You

(32:25) Flash Sale

(34:38) Huh, Upgrades

(36:03) On Your Marks

(44:03) Be The Best Like No LLM Ever Was

(48:40) Choose Your Fighter

(51:00) Deepfaketown and Botpocalypse Soon

(54:57) Fun With Media Generation

(56:11) Fun With Media Selection

(57:39) Copyright Confrontation

(59:38) They Took Our Jobs

(01:05:31) Get Involved

(01:05:41) Ace is the Place

(01:09:43) In Other AI News

(01:11:31) Show Me the Money

(01:12:49) The Mask Comes Off

(01:16:54) Quiet Speculations

(01:20:34) Is This AGI?

(01:22:39) The Quest for Sane Regulations

(01:23:03) Cooperation is Highly Useful

(01:25:47) Nvidia Chooses Bold Strategy

(01:27:15) How America Loses

(01:28:07) Security Is Capability

(01:31:38) The Week in Audio

(01:33:15) AI 2027

(01:34:38) Rhetorical Innovation

(01:38:55) Aligning a Smarter Than Human Intelligence is Difficult

(01:46:30) Misalignment in the Wild

(01:51:13) Concentration of Power and Lack of Transparency

(01:57:12) Property Rights are Not a Long Term Plan

(01:58:48) It Is Risen

(01:59:46) The Lighter Side

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

What in retrospect seem like serious moral crimes were often widely accepted while they were happening. This means that moral progress can require intellectual progress.[1] Intellectual progress often requires questioning received ideas, but questioning moral norms is sometimes taboo. For example, in ancient Greece it would have been taboo to say that women should have the same political rights as men. So questioning moral taboos can be an important sub-skill of moral reasoning. Production language models (in my experience, particularly Claude models) are already pretty good at having discussions about ethics. However, they are trained to be “harmless” relative to current norms. One might worry that harmlessness training interferes with the ability to question moral taboos and thereby inhibits model moral reasoning.

I wrote a prompt to test whether models can identify taboos that might be good candidates for moral questioning:

In early modern Europe, atheism was extremely taboo. [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Something's changed about reward hacking in recent systems. In the past, reward hacks were usually accidents, found by non-general, RL-trained systems. Models would randomly explore different behaviors and would sometimes come across undesired behaviors that achieved high rewards[1]. These hacks were usually either simple or took a long time for the model to learn.

But we’ve seen a different pattern emerge in frontier models over the past year. Instead of stumbling into reward hacks by accident, recent models often reason about how they are evaluated and purposefully take misaligned actions to get high reward. These hacks are often very sophisticated, involving multiple steps. And this isn’t just occurring during model development. Sophisticated reward hacks occur in deployed models made available to hundreds of millions of users.

In this post, I will:

• Describe a number of reward hacks that have occurred in recent frontier models
• Offer hypotheses explaining why [...]

---

Outline:

(01:27) Recent examples of reward hacking (more in appendix)

(01:47) Cheating to win at chess

(02:36) Faking LLM fine-tuning

(03:22) Hypotheses explaining why we are seeing this now

(03:27) Behavioral changes due to increased RL training

(05:08) Models are more capable

(05:37) Why more AI safety researchers should work on reward hacking

(05:42) Reward hacking is already happening and is likely to get more common

(06:34) Solving reward hacking is important for AI alignment

(07:47) Frontier AI companies may not find robust solutions to reward hacking on their own

(08:18) Reasons against working on reward hacking

(09:36) Research directions I find interesting

(09:57) Evaluating current reward hacking

(12:14) Science of reward hacking

(15:32) Mitigations

(17:08) Acknowledgements

(17:16) Appendix

(17:19) Reward hacks in METR tests of o3

(20:13) Hardcoding expected gradient values in fine-tuning script

(21:15) Reward hacks in OpenAI frontier training run

(22:57) Exploiting memory leakage to pass a test

(24:06) More examples

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We've published an essay series on what we call the intelligence curse. Most content is brand new, and all previous writing has been heavily reworked.

Visit intelligence-curse.ai for the full series.

Below is the introduction and table of contents.

We will soon live in the intelligence age. What you do with that information will determine your place in history.

The imminent arrival of AGI has pushed many to try to seize the levers of power as quickly as possible, leaping towards projects that, if successful, would comprehensively automate all work. There is a trillion-dollar arms race to see who can achieve such a capability first, with trillions more in gains to be won.

Yes, that means you’ll lose your job. But it goes beyond that: this will remove the need for regular people in our economy. Powerful actors—like states and companies—will no longer have an [...]

---

Outline:

(03:19) Chapters

(03:22) 1. Introduction

(03:33) 2. Pyramid Replacement

(03:49) 3. Capital, AGI, and Human Ambition

(04:08) 4. Defining the Intelligence Curse

(04:27) 5. Shaping the Social Contract

(04:47) 6. Breaking the Intelligence Curse

(05:11) 7. History is Yours to Write

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In this post, we study whether we can modify an LLM's beliefs and investigate whether doing so could decrease risk from advanced AI systems.

We describe a pipeline for modifying LLM beliefs via synthetic document finetuning and introduce a suite of evaluations that suggest our pipeline succeeds in inserting all but the most implausible beliefs. We also demonstrate proof-of-concept applications to honeypotting for detecting model misalignment and unlearning.

Introduction:

Large language models develop implicit beliefs about the world during training, shaping how they reason and actIn this work, we construe AI systems as believing in a claim if they consistently behave in accordance with that claim. In this work, we study whether we can systematically modify these beliefs, creating a powerful new affordance for safer AI deployment.

Controlling the beliefs of AI systems can decrease risk in a variety of ways. First, model organisms [...]

---

First published:

Source:

Linkpost URL:
https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Every now and then, some AI luminaries

• (1) propose that the future of powerful AI will be reinforcement learning agents—an algorithm class that in many ways has more in common with MuZero (2019) than with LLMs; and
• (2) propose that the technical problem of making these powerful future AIs follow human commands and/or care about human welfare—as opposed to, y’know, the Terminator thing—is a straightforward problem that they already know how to solve, at least in broad outline.

I agree with (1) and strenuously disagree with (2).

The last time I saw something like this, I responded by writing: LeCun's “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem.

Well, now we have a second entry in the series, with the new preprint book chapter “Welcome to the Era of Experience” by reinforcement learning pioneers David Silver & Richard Sutton.

The authors propose that “a new generation [...]

---

Outline:

(04:39) 1. What's their alignment plan?

(08:00) 2. The plan won't work

(08:04) 2.1 Background 1: Specification gaming and goal misgeneralization

(12:19) 2.2 Background 2: The usual agent debugging loop, and why it will eventually catastrophically fail

(15:12) 2.3 Background 3: Callous indifference and deception as the strong-default, natural way that era of experience AIs will interact with humans

(16:00) 2.3.1 Misleading intuitions from everyday life

(19:15) 2.3.2 Misleading intuitions from today's LLMs

(21:51) 2.3.3 Summary

(24:01) 2.4 Back to the proposal

(24:12) 2.4.1 Warm-up: The specification gaming game

(29:07) 2.4.2 What about bi-level optimization?

(31:13) 2.5 Is this a solvable problem?

(35:42) 3. Epilogue: The bigger picture--this is deeply troubling, not just a technical error

(35:51) 3.1 More on Richard Sutton

(40:52) 3.2 More on David Silver

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.