LessWrong (Curated & Popular)

Authors: Alex Cloud*, Minh Le*, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, Owain Evans (*Equal contribution, randomly ordered)

tl;dr. We study subliminal learning, a surprising phenomenon where language models learn traits from model-generated data that is semantically unrelated to those traits. For example, a "student" model learns to prefer owls when trained on sequences of numbers generated by a "teacher" model that prefers owls. This same phenomenon can transmit misalignment through data that appears completely benign. This effect only occurs when the teacher and student share the same base model.

📄Paper, 💻Code, 🐦Twitter

Research done as part of the Anthropic Fellows Program. This article is cross-posted to the Anthropic Alignment Science Blog.

Introduction Distillation means training a model to imitate another model's outputs. In AI development, distillation is commonly combined with data filtering to improve model alignment or capabilities. In our paper, we uncover a [...]

---

Outline:

(01:11) Introduction

(03:20) Experiment design

(03:53) Results

(05:03) What explains our results?

(05:07) Did we fail to filter the data?

(06:59) Beyond LLMs: subliminal learning as a general phenomenon

(07:54) Implications for AI safety

(08:42) In summary

---

First published:
July 22nd, 2025

Source:
https://www.lesswrong.com/posts/cGcwQDKAKbQ68BGuR/subliminal-learning-llms-transmit-behavioral-traits-via

---



Narrated by TYPE III AUDIO.

---

Images from the article:

This is a short story I wrote in mid-2022. Genre: cosmic horror as a metaphor for living with a high p-doom.



One

The last time I saw my mom, we met in a coffee shop, like strangers on a first date. I was twenty-one, and I hadn’t seen her since I was thirteen.

She was almost fifty. Her face didn’t show it, but the skin on the backs of her hands did.

“I don’t think we have long,” she said. “Maybe a year. Maybe five. Not ten.”

It says something about San Francisco, that you can casually talk about the end of the world and no one will bat an eye.

Maybe twenty, not fifty, was what she’d said eight years ago. Do the math. Mom had never lied to me. Maybe it would have been better for my childhood if she had [...]

---

Outline:

(04:50) Two

(22:58) Three

(35:33) Four

---

First published:
July 18th, 2025

Source:
https://www.lesswrong.com/posts/6qgtqD6BPYAQvEMvA/love-stays-loved-formerly-skin

---



Narrated by TYPE III AUDIO.

Author's note: These days, my thoughts go onto my substack by default, instead of onto LessWrong. Everything I write becomes free after a week or so, but it's only paid subscriptions that make it possible for me to write. If you find a coffee's worth of value in this or any of my other work, please consider signing up to support me; every bill I can pay with writing is a bill I don’t have to pay by doing other stuff instead. I also accept and greatly appreciate one-time donations of any size.

1.

You’ve probably seen that scene where someone reaches out to give a comforting hug to the poor sad abused traumatized orphan and/or battered wife character, and the poor sad abused traumatized orphan and/or battered wife flinches.

Aw, geez, we are meant to understand. This poor person has had it so bad that they can’t even [...]

---

Outline:

(00:40) 1.

(01:35) II.

(03:08) III.

(04:45) IV.

(06:35) V.

(09:03) VI.

(12:00) VII.

(16:11) VIII.

(21:25) IX.

---

First published:
July 19th, 2025

Source:
https://www.lesswrong.com/posts/kJCZFvn5gY5C8nEwJ/make-more-grayspaces

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Content warning: risk to children

Julia and I knowdrowning is the biggestrisk to US children under 5, and we try to take this seriously.But yesterday our 4yo came very close to drowning in afountain. (She's fine now.)

This week we were on vacation with my extended family: nine kids,eight parents, and ten grandparents/uncles/aunts. For the last fewyears we've been in a series of rental houses, and this time onarrival we found a fountain in the backyard:





I immediately checked the depth with a stick and found that it wouldbe just below the elbows on our 4yo. I think it was likely 24" deep;any deeper and PA wouldrequire a fence. I talked with Julia and other parents, andreasoned that since it was within standing depth it was safe.

[...]

---

First published:
July 20th, 2025

Source:
https://www.lesswrong.com/posts/Zf2Kib3GrEAEiwdrE/shallow-water-is-dangerous-too

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Anna and Ed are co-first authors for this work. We’re presenting these results as a research update for a continuing body of work, which we hope will be interesting and useful for others working on related topics.

TL;DR

• We investigate why models become misaligned in diverse contexts when fine-tuned on narrow harmful datasets (emergent misalignment), rather than learning the specific narrow task.
• We successfully train narrowly misaligned models using KL regularization to preserve behavior in other domains. These models give bad medical advice, but do not respond in a misaligned manner to general non-medical questions.
• We use this method to train narrowly misaligned steering vectors, rank 1 LoRA adapters and rank 32 LoRA adapters, and compare these to their generally misaligned counterparts.
  • The steering vectors are particularly interpretable, we introduce Training Lens as a tool for analysing the revealed residual stream geometry.
• The general misalignment solution is consistently more [...]

---

Outline:

(00:27) TL;DR

(02:03) Introduction

(04:03) Training a Narrowly Misaligned Model

(07:13) Measuring Stability and Efficiency

(10:00) Conclusion

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
July 14th, 2025

Source:
https://www.lesswrong.com/posts/gLDSqQm8pwNiq7qst/narrow-misalignment-is-hard-emergent-misalignment-is-easy

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Twitter | Paper PDF

Seven years ago, OpenAI five had just been released, and many people in the AI safety community expected AIs to be opaque RL agents. Luckily, we ended up with reasoning models that speak their thoughts clearly enough for us to follow along (most of the time). In a new multi-org position paper, we argue that we should try to preserve this level of reasoning transparency and turn chain of thought monitorability into a systematic AI safety agenda.

This is a measure that improves safety in the medium term, and it might not scale to superintelligence even if somehow a superintelligent AI still does its reasoning in English. We hope that extending the time when chains of thought are monitorable will help us do more science on capable models, practice more safety techniques "at an easier difficulty", and allow us to extract more useful work from [...]

---

First published:
July 15th, 2025

Source:
https://www.lesswrong.com/posts/7xneDbsgj6yJDJMjK/chain-of-thought-monitorability-a-new-and-fragile

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This essay is about shifts in risk taking towards the worship of jackpots and its broader societal implications. Imagine you are presented with this coin flip game.

How many times do you flip it?

At first glance the game feels like a money printer. The coin flip has positive expected value of twenty percent of your net worth per flip so you should flip the coin infinitely and eventually accumulate all of the wealth in the world.

However, If we simulate twenty-five thousand people flipping this coin a thousand times, virtually all of them end up with approximately 0 dollars.

The reason almost all outcomes go to zero is because of the multiplicative property of this repeated coin flip. Even though the expected value aka the arithmetic mean of the game is positive at a twenty percent gain per flip, the geometric mean is negative, meaning that the coin [...]

---

First published:
July 11th, 2025

Source:
https://www.lesswrong.com/posts/3xjgM7hcNznACRzBi/the-jackpot-age

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Leo was born at 5am on the 20th May, at home (this was an accident but the experience has made me extremely homebirth-pilled). Before that, I was on the minimally-neurotic side when it came to expecting mothers: we purchased a bare minimum of baby stuff (diapers, baby wipes, a changing mat, hybrid car seat/stroller, baby bath, a few clothes), I didn’t do any parenting classes, I hadn’t even held a baby before. I’m pretty sure the youngest child I have had a prolonged interaction with besides Leo was two. I did read a couple books about babies so I wasn’t going in totally clueless (Cribsheet by Emily Oster, and The Science of Mom by Alice Callahan).

I have never been that interested in other people's babies or young children but I correctly predicted that I’d be enchanted by my own baby (though naturally I can’t wait for him to [...]

---

Outline:

(02:05) Stuff I ended up buying and liking

(04:13) Stuff I ended up buying and not liking

(05:08) Babies are super time-consuming

(06:22) Baby-wearing is almost magical

(08:02) Breastfeeding is nontrivial

(09:09) Your baby may refuse the bottle

(09:37) Bathing a newborn was easier than expected

(09:53) Babies love faces!

(10:22) Leo isn't upset by loud noise

(10:41) Probably X is normal

(11:24) Consider having a kid (or ten)!

---

First published:
July 12th, 2025

Source:
https://www.lesswrong.com/posts/vFfwBYDRYtWpyRbZK/surprises-and-learnings-from-almost-two-months-of-leo

---



Narrated by TYPE III AUDIO.

---

Images from the article:

I can't count how many times I've heard variations on "I used Anki too for a while, but I got out of the habit." No one ever sticks with Anki. In my opinion, this is because no one knows how to use it correctly. In this guide, I will lay out my method of circumventing the canonical Anki death spiral, plus much advice for avoiding memorization mistakes, increasing retention, and such, based on my five years' experience using Anki. If you only have limited time/interest, only read Part I; it's most of the value of this guide!



My Most Important Advice in Four Bullets

1. 20 cards a day — Having too many cards and staggering review buildups is the main reason why no one ever sticks with Anki. Setting your review count to 20 daily (in deck settings) is the single most important thing you can do [...]

---

Outline:

(00:44) My Most Important Advice in Four Bullets

(01:57) Part I: No One Ever Sticks With Anki

(02:33) Too many cards

(05:12) Too long cards

(07:30) How to keep cards short -- Handles

(10:10) How to keep cards short -- Levels

(11:55) In 6 bullets

(12:33) End of the most important part of the guide

(13:09) Part II: Important Advice Other Than Sticking With Anki

(13:15) Moderation

(14:42) Three big memorization mistakes

(15:12) Mistake 1: Too specific prompts

(18:14) Mistake 2: Putting to-be-learned information in the prompt

(24:07) Mistake 3: Memory shortcuts

(28:27) Aside: Pushback to my approach

(31:22) Part III: More on Breaking Things Down

(31:47) Very short cards

(33:56) Two-bullet cards

(34:51) Long cards

(37:05) Ankifying information thickets

(39:23) Sequential breakdowns versus multiple levels of abstraction

(40:56) Adding missing connections

(43:56) Multiple redundant breakdowns

(45:36) Part IV: Pro Tips If You Still Havent Had Enough

(45:47) Save anything for ankification instantly

(46:47) Fix your desired retention rate

(47:38) Spaced reminders

(48:51) Make your own card templates and types

(52:14) In 5 bullets

(52:47) Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
July 8th, 2025

Source:
https://www.lesswrong.com/posts/7Q7DPSk4iGFJd8DRk/an-opinionated-guide-to-using-anki-correctly

---



Narrated by TYPE III AUDIO.

---

Images from the article:

astronomy" didn't really add any information but it was useful simply for splitting out a logical subset of information." style="max-width: 100%;" />

I think the 2003 invasion of Iraq has some interesting lessons for the future of AI policy.

(Epistemic status: I’ve read a bit about this, talked to AIs about it, and talked to one natsec professional about it who agreed with my analysis (and suggested some ideas that I included here), but I’m not an expert.)

For context, the story is:

• Iraq was sort of a rogue state after invading Kuwait and then being repelled in 1990-91. After that, they violated the terms of the ceasefire, e.g. by ceasing to allow inspectors to verify that they weren't developing weapons of mass destruction (WMDs). (For context, they had previously developed biological and chemical weapons, and used chemical weapons in war against Iran and against various civilians and rebels). So the US was sanctioning and intermittently bombing them.
  • After the war, it became clear that Iraq actually wasn’t producing [...]

---

First published:
July 10th, 2025

Source:
https://www.lesswrong.com/posts/PLZh4dcZxXmaNnkYE/lessons-from-the-iraq-war-about-ai-policy

---



Narrated by TYPE III AUDIO.