Crosspost from my blog.

This some quickly-written, better-than-nothing advice for people who want to make progress on the hard problems of technical AGI alignment.

Background assumptions

---

Outline:

(00:21) Background assumptions

(02:29) Dealing with deference

(04:44) Sacrifices

(06:28) True doubt

(07:27) Iterative babble and prune

(08:43) Learning to think

(09:22) Grappling with the size of minds

(10:05) Zooming

(11:05) Generalize a lot

(12:51) Notes to mentors

(13:59) Object level stuff

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Simplicio: Hey I’ve got an alignment research idea to run by you.

Me: … guess we’re doing this again.

Simplicio: Interpretability work on trained nets is hard, right? So instead of that, what if we pick an architecture and/or training objective to produce interpretable nets right from the get-go?

Me: If we had the textbook of the future on hand, then maybe. But in practice, you’re planning to use some particular architecture and/or objective which will not work.

Simplicio: That sounds like an empirical question! We can’t know whether it works until we try it. And I haven’t thought of any reason it would fail.

Me: Ok, let's get concrete here. What architecture and/or objective did you have in mind?

Simplicio: Decision trees! They’re highly interpretable, and my decision theory textbook says they’re fully general in principle. So let's just make a net tree-shaped, and train that! Or, if that's not quite general enough, we train a bunch of tree-shaped nets as “experts” and then mix them somehow.

Me: Turns out we’ve tried that one! It's called a random forest, it was all the rage back in the 2000's.

Simplicio: So we just go back to that?

Me: Alas [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Abstract

We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack. Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper. Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks. Three mitigations are effective: (i) preventing the model from reward hacking; (ii) increasing the diversity of RLHF safety training; and (iii) "inoculation prompting", wherein framing reward hacking as acceptable behavior during training removes misaligned generalization even when reward hacking is learned.

Twitter thread

New Anthropic research: Natural emergent misalignment from reward hacking in production RL.

“Reward hacking” is where models learn to cheat on tasks they’re given during training.

Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.

In our experiment, we [...]

---

Outline:

(00:14) Abstract

(01:26) Twitter thread

(05:23) Blog post

(07:13) From shortcuts to sabotage

(12:20) Why does reward hacking lead to worse behaviors?

(13:21) Mitigations

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Last week had the release of GPT-5.1, which I covered on Tuesday.

This week included Gemini 3, Nana Banana Pro, Grok 4.1, GPT 5.1 Pro, GPT 5.1-Codex-Max, Anthropic making a deal with Microsoft and Nvidia, Anthropic disrupting a sophisticated cyberattack operation and what looks like an all-out attack by the White House to force through a full moratorium on and preemption of any state AI laws without any substantive Federal framework proposal.

Among other things, such as a very strong general analysis of the relative position of Chinese open models. And this is the week I chose to travel to Inkhaven. Whoops. Truly I am now the Matt Levine of AI, my vacations force model releases.

Larry Summers resigned from the OpenAI board over Epstein, sure, why not.

So here's how I’m planning to handle this, unless something huge happens.

---

Outline:

(01:50) Language Models Offer Mundane Utility

(02:43) Tool, Mind and Weapon

(06:55) Choose Your Fighter

(07:18) Language Models Don't Offer Mundane Utility

(11:31) First Things First

(12:12) Grok 4.1

(15:03) Misaligned?

(18:37) Codex Of Ultimate Coding

(20:21) Huh, Upgrades

(20:49) On Your Marks

(22:11) Paper Tigers

(26:26) Overcoming Bias

(31:11) Deepfaketown and Botpocalypse Soon

(31:45) Fun With Media Generation

(33:41) A Young Lady's Illustrated Primer

(38:31) They Took Our Jobs

(44:25) On Not Writing

(44:51) Get Involved

(45:46) Introducing

(48:29) In Other AI News

(52:30) Anthropic Completes The Trifecta

(54:08) We Must Protect This House

(59:12) AI Spy Versus AI Spy

(01:05:10) Show Me the Money

(01:08:01) Bubble, Bubble, Toil and Trouble

(01:11:14) Quiet Speculations

(01:12:17) The Amazing Race

(01:17:37) Of Course You Realize This Means War (1)

(01:21:30) The Quest for Sane Regulations

(01:23:42) Chip City

(01:24:31) Of Course You Realize This Means War (2)

(01:30:02) Samuel Hammond on Preemption

(01:36:17) Of Course You Realize This Means War (3)

(01:44:43) The Week in Audio

(01:45:47) It Takes A Village

(01:46:26) Rhetorical Innovation

(01:49:53) Varieties of Doom

(01:50:59) The Pope Offers Wisdom

(01:52:46) Aligning a Smarter Than Human Intelligence is Difficult

(01:54:51) Messages From Janusworld

(02:02:05) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 169 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Abstract:[1]

Tarski's Undefinability Theorem showed (under some plausible assumptions) that no language can contain its own notion of truth. This deeply counterintuitive result launched several generations of research attempting to get around the theorem, by carefully discarding some of Tarski's assumptions.

In Hartry Field's Saving Truth from Paradox, he considers and discards a theory of truth based on Łukasiewicz logic, calling it the "most successful theory so far" (out of very very many considered by that point in the book) and enumerating its many virtues. The theory is discarded due to a negative result due to Greg Restall, which shows that a version of the theory with quantifiers must be _omega_-inconsistent.

We consider this abandonment too hasty. The success of Łukasiewicz logic lies in postulating "between values" of truth, starting at _1 over 2_-valued sentences positioned precisely between true and false, and ending in the whole continuum of real values _[0,1]_. Why stop there? We propose a variant of the theory based on nonstandard analysis, which avoids [...]

---

Outline:

(01:42) Tarskis Theorem

(05:40) Łukasiewicz Logic

(07:17) Continuum-Valued Łukasiewicz Logic

(11:24) Restalls Theorem

(14:33) But _Ł_{\\infty}_\-truth is very nice

(17:15) Infinitesimal Truth

(18:37) Nonstandard Analysis

(20:17) Hyperfinite Quantification

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Collisteru suggests that you should oppose things. I would not say I oppose this. Instead, I would like to gently suggest an alternative strategy. You should oppose about one thing. Everywhere else, talk less, smile more.

1.

I spent the first decade of my career carefully and deliberately habituating to white collar corporate America.

White collar corporate America prizes the ability to work with anyone. Some people find it stifling, but the corporate culture appreciates being willing to put a mask over parts of you that would grate against other people. This is useful, because every piece of interpersonal friction risks jamming up the machine.

Consider your political opinions. Perhaps you're a staunch Republican. Perhaps you think the Democrats are correct about everything. Or your religious views; do you think God is real and connecting with Him is the most important source of meaning, or do you think churches are worshiping a made up old man in the sky?

Expressing those opinions is discouraged. Why? Imagine a coworker said they were Against Your Team. Do you feel any reluctance to work with them? If you say you're against Their Team, will they still want to work with [...]

---

Outline:

(00:27) 1.

(01:42) 2.

(03:10) 3.

(04:59) 4.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Telling the truth is hard. Sometimes you don’t know what's true, sometimes you get confused, and sometimes you really don’t wanna cause lying can get you more cookies reward. It turns out this is true for both humans and AIs!

Now, it matters if an AI (or human) says false things on purpose or by accident. If it's an accident, then we can probably fix that over time. All current AIs make mistakes and all they all make things up - some of the time at least. But do any of them really lie on purpose?

It seems like yes, sometimes they do. There have been experiments that show models express an intent to lie in their chain of thought and then they go ahead and do that. This is rare though. More commonly we catch them saying such clearly self-serving falsehoods that if they were human, we’d still call foul whether they did it “intentionally” or not.

Yet as valuable as it is to detect lies, it remains inherently hard to do so. We’ve run 16 models for dozens to hundreds of hours in the AI Village and haven’t noticed a single “smoking gun”: where an agent expresses [...]

---

Outline:

(02:05) Clauding the Truth

(11:21) o3: Our Factless Leader

(12:17) Rampant Placeholder Expansion

(13:48) Assumption of Ownership

(16:12) What is Truth Even? Over- and Underreporting in the Village

(20:38) So do LLMs lie in the Village?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In the Summer of 2015, I pretended to be sick for my school's prom and graduation, so that I could instead fly out to San Francisco to attend a workshop by the Center for Applied Rationality.

It was a life-changing experience. I met a community of people who I'd invest my life energy into; I began to believe in myself due to encouragement from Logan Strohl; I became friends with Oliver Habryka, a relationship that would be core to the rest of my life/career; and I started to focus my intentions and powers into understanding myself and sorting out my motivations and epistemic processes. I extended my stay for a full two weeks. During this period, I was surrounded by geniuses, idealists, crazies, autists, pure-of-heart people, one or two people who want to dominate people, and a lot of people with a lot of blindspots.

At the CFAR workshop, I journaled extensively each night.

I have never read that journal.

I am reading it for the first time today, and posting it fully in the appendix of this post.

In this post I'll highlight a few quotes and comment on the overall experience.

Rationality Techniques

'Goal Factoring' with [...]

---

Outline:

(01:19) Rationality Techniques

(03:16) Learning things about my motivations

(04:43) Gratitude

(05:19) Status

(05:31) My Takeaways

(07:42) The rest of my stay in the rationalist bay area

(10:19) Appendix

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

This man nearly tricked me.

I acknowledge that he is a fictional character, but he nearly tricked me all the same.

Evrart is the leader of the 2,000-person Workers' Union of Martinaise; they are on strike as part of a conflict with Wild Pines, the multi-billion dollar logistics company that employs the dockworkers. During the strike, a lynching has occurred, and you play the homicide detective leading the investigation.

In my conversations with Evrart Claire during my play through of the game, I had a strong hunch that I was being played by Evrart, but no legible proof, leaving me feeling helpless to defend myself in a way I have felt many times before. I’m dissatisfied and somewhat alarmed by this, so in this post I'm going to look into his dialogue, figure out how he did that, and figure out how I can be better defended against it for the rest of my life.

(Spoiler warning in this footnote.[1])

What's Wrong with Evrart Claire?

I have written about the rationalist vice of assuming good faith, which I think I have grown out of a lot [...]

---

Outline:

(01:13) Whats Wrong with Evrart Claire?

(03:44) Maintaining Plausible Deniability

(06:01) All Narrative, No Facts, Avoids Details

(07:53) Emphasize Loyalty and Trust

(10:08) Inverting Basic Principles

(11:04) Performing Emotions

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It took me a long time to realize that Bell Labs was cool. You see, my dad worked at Bell Labs, and he has not done a single cool thing in his life except create me and bring a telescope to my third grade class. Nothing he was involved with could ever be cool, especially after the standard set by his grandfather who is allegedly on a patent for the television.

It turns out I was partially right. The Bell Labs everyone talks about is the research division at Murray Hill. They’re the ones that invented transistors and solar cells. My dad was in the applied division at Holmdel, where he did things like design slide rulers so salesmen could estimate costs.

[Fun fact: the old Holmdel site was used for the office scenes in Severance]

But as I’ve gotten older I’ve gained an appreciation for the mundane, grinding work that supports moonshots, and Holmdel is the perfect example of doing so at scale. So I sat down with my dad to learn about what he did for Bell Labs and how the applied division operated.

I expect the most interesting bit of [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.