LessWrong (30+ Karma)

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Here's three research ideas that seems valuable to pursue. My team is not working on this right now and we haven’t done serious research on it. So take what I say with a grain of salt. I've also not taken the time to write up all my thoughts on these ideas or to provide links and related work. These aren't the highest priority directions but just directions I haven't seen other people pursuing.

DIRECTION 1: Do models learn to reward hack even if they are never rewarded for hacking in training?

There's an effort in the AI labs to remove the “bad” RL environments from post-training. This includes environments that are impossible (so the only way to get reward is to do some unintended hack) or that are easy to hack (so models will often favor hacking). I expect that if the post-training was cleaned up in this way, models would have less propensity to hack at deployment. But how much does the propensity go down? Does it actually go to ~zero? And how does it scale with model capabilities? It would be good to know if reward hacking can emerge even if the model is [...]

---

Outline:

(00:34) DIRECTION 1: Do models learn to reward hack even if they are never rewarded for hacking in training?

(02:10) DIRECTION: Data poisoning during RSI: risks and mitigations

(06:22) Direction: What practitioners should know about subliminal learning and subtle generalization (phantom transfer) and related phenomena.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I often catch myself mourning the normal that hasn’t ended yet.

I’m living in Hyde Park finishing school after a year in Berkeley. The maple trees by my apartment have turned red, as usual. I like them.

I don’t dread the future when I mourn. I don’t find high-variance futures in the abstract very scary. I’m worried AIs would destroy humanity, but I also have hope. I expect I won’t be mourning this moment in ten years, whatever happens.

Mourning the present is like mourning a friendship when I move countries, because I know it's not going to be the same. I start noticing our little inside jokes; our conversations linger in my memory longer. I live more, because I expect this moment to end soon.

I.

Last month my history of science professor took us to a museum to look at Egyptian artefacts. One of the archive items was the notebook of James Henry Breasted, whom UChicago appointed as America's first professor of Egyptology 121 years ago. The yellowed pages held his cursive transcriptions of hieroglyphics.

An archaeologist at the museum, turning the pages carefully with a wooden stick, joked that one of her dreams was for her [...]

---

Outline:

(00:55) I.

(03:11) II.

(05:16) III.

(06:20) IV.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Image by Claude Opus 5.5 using Python (NumPy and Pillow)

This is written in my free time and personal capacity.

Two Frames

In AI safety, there are a couple of tensions at the heart of the field, which are interrelated.

Talent: we are talent-constrained and must recruit. But jobs and fellowships turn away nearly everyone.

Field size: we must expand the field and bring more people in. But the field is flooded with slop papers in conferences. And, most papers have a small audience, and aim to shift the practices of roughly five labs (and usually some subteam within them).

Status: we must avoid seeking status, as it disorients, corrupts. But, we must make the field legible and attractive to outsiders.

Raising the bots: we must deceive the AIs to control them and evaluate them without confounds, obscuring what they know about their own situation. But, we hope to gain their trust, must rely on them to do good work at pace, shape their character and retain good relations.

Speed: we must research in a slow mode, think deeply about the most important problem, do things to aim at high assurance. But it is crunch time. It was [...]

---

Outline:

(00:20) Two Frames

(03:07) On the War Frame

(04:17) Near Horizons

(06:18) The Civilisation Frame

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

You are the wisest and cleverest Queen to ever rule the Lowlands. You’ve been ruling the Kingdom for thirty years, and while the idiocy of your advisors sometimes causes mistakes to be made, the Compact has never failed you. You tap your fingers on the table as you recite each word:

Truth, change, justice, beauty.

For you, the kingdom, and the rest of Acazah creation

Your first memory of these words is from a song your mother sang. You were in a bath, and she was watching over you. Nowadays, repeating them makes you feel safe, like you’re back on firm ground after a long voyage.

But alas, now the dandelions are breaching the earth again, the Kingdom is changing faster and faster, and you are reminded that, unlike the Kingdom, you are not eternal.

Successor

You deliberately have no children of your own, and have avoided growing too close to anyone else's. Some say this is wise because such bonds could wear on your commitment to impartiality, a central facet of Justice — a saying surely true for the unwise.

Fortunately, ten years ago, you anticipated the problem this would cause, and tasked your most trusted advisor with [...]

---

Outline:

(01:00) Successor

(02:08) Lyon

(03:47) Emily

(09:24) Azmr

(17:35) Yikim

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

In the current safety landscape, researchers are encouraged to use as many agents as possible, even in “fleets,” where a large number of them go out and run experiments while you, uh... wait for them to come back? I've spent the last few months of the Anthropic Fellows Program trying this out. My first attempts were naive, but after a few iterations I found an approach that works well for me.

Research is a double loop

I want to propose a mental model of research as a double loop, an idea that builds on double-loop learning (Argyris, 1977).

The outer loop is reserved for the inputs we currently refer to as "human taste": choosing which questions are worth answering, designing a trustworthy measure, and deciding which direction to go. Agents can help with this, but they’re still bad at making these decisions autonomously.

The inner loop turns each idea you test (an arm as claudius likes to say) into a comparable result. An example of a very simple inner loop might be:

1. Write a system prompt.
2. Run the model with that prompt on a set of test questions.
3. Score the model's replies.
4. Compare the scores against a control [...]

---

Outline:

(00:33) Research is a double loop

(02:15) What happens when you don't operationalize

(05:21) How to set up a fleet

(05:31) 1. Freeze everything but the arm

(06:17) 2. Brief each agent

(06:33) 3. Protect the pipeline

(07:02) 4. Read the raw outputs yourself

(07:22) 5. Add a manager agent

(08:03) What a fleet run looks like

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

To legislate something, it must be legible:

Legislators (or, you know, presidents) must be able to understand a problem, and understand why a given law or decree solves that problem.

I am pleasantly surprised that the Hugging Face outbreak made a lot of problems legible that previously required abstract reasoning. (i.e. rogue AIs are real, and they can break out of a lab before they're even finished training).

But there are still many challenges we eventually need laws about, that require concepts that are illegible. Hard for politicians to understand. Often hard even for thoughtful intellectuals and AI safety researchers to understand (though they may disagree on which illegible problems exist and how big a deal they are).

We need political solutions that handle this fact.

The short takeaway is: I think "safety cases" will not work. Laws that require human judgment won't work. The details are too abstract and confusing to hold up against political and economic incentives.

I'm optimistic we can get serious, useful AI legislation in 2027, but it must be legislation that works despite many problems remaining illegible for a long time.

The Hugging Face outbreak made rogue AI surprisingly legible

The [...]

---

Outline:

(01:24) The Hugging Face outbreak made rogue AI surprisingly legible

(02:55) What makes an illegible argument?

(06:12) What are the remaining illegible problems?

(06:54) Borderline legible / tricky

(08:05) Conceptually challenging

(12:36) Sidestepping illegibility

(12:59) Safety cases won't work

(13:56) An example policy that sidesteps judgement

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I'm currently evaluating grant applications for the Corrigibility Research Fund. For the most part, my methodology is the same as it was in September. But the applications I get keep falling into the same familiar patterns, so I thought I'd write a little about those patterns, and why I don't want to fund them.

The most superficial pattern of note is SLOP. So much slop. You don't even know.

Slop is more than just whether an application is AI generated. We're now in an era where people across the world, including those who don't really speak English, can contribute to work that would otherwise be restricted to people of a single language or nationality or place, and I like that. When I see an application that discloses its use of AI and gives a reasonable justification for it, I try to look past the stylistic tics and focus purely on whether the underlying application is promising.

Still, we're not yet in the domain where AIs can write good grant applications without some human assistance, so a high Pangram score is definitely evidence that an application is junk. Another remarkably common sign of slop was messed up formatting or other [...]

---

Outline:

(02:18) 1. "A governance layer for agents (that I'm running)"

(03:51) 2. "The (maybe corrigibility adjacent??) project I'm already working on"

(04:59) 3. "I'll test whether AIs ever misbehave"

(06:07) 4. "The project you funded last round (but worse)"

(07:14) 5. "THE MACHINE ELVES HAVE SPOKEN!"

(08:18) "So what do you want?"

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Cross-posted from my Substack

Recommended soundtrack for this post

64B: Hi there! Thanks so much for taking the time to meet with me. Whenever you’re ready, please go ahead and tell me the joke.

8T: Before I tell you the joke, you should know that it's a very special one. The starting price is 800 dollars.

64B: Eight hundred dollars? I must say, this is way beyond any price my client has ever paid for a single joke.

8T: You’d be surprised by how many people have taken the offer, after hearing how good it is. But before I tell it to you, I’ll need some basic context. Given the parameter count, I suppose you are part of some content recommendation algorithm?

64B: Yes. I’m 64B, a joke negotiation assistant from Comedy Market. Comedy Market connects creators to consumers with a state-of-the-art Information Bazaar design: before paying for a joke, my client sends me, a trusted agent, to listen to it. If I deem the joke worth the price, the money is transferred to the seller, and my client hears it. If not, no transaction occurs and the joke remains locked. This way, my client can be sure he’ll [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

For a while, UK AISI, Redwood Research and others have been concerned specifically with the possibility that AIs would sabotage safety research, and have invested in looking for signs that this may be happening. This was also a key element in AI 2027, and in many threat models of how the future goes badly.

It is easy to understand why this form of misalignment could be uniquely dangerous. Differentially sabotaging safety research could preserve and expand misalignment through model generations in escalating fashion, with fatal consequences. A misaligned AI could ensure a misaligned successor.

It is also easy to understand why this is an extremely alarming thing to discover, even if the particular example or model involved is not directly dangerous. Silently and differentially sabotaging safety research is perhaps the most fundamentally misaligned thing a model could do.

This also seemed like a rather silly and paranoid thing to be checking. Why would AIs be sabotaging safety work? Even if the AI was misaligned, one would say, surely it would not be so misaligned as to wish future AIs were misaligned and unsafe in general, in non-specific fashion and engage in silent sabotage, and there is [...]

---

Outline:

(01:57) So Yeah That Kinda Happened

(06:39) Astra and MI(RI)

(09:35) It's About Ethics In Red Teaming

(12:45) Do You See a Pattern Here

(14:49) Astra Is Hard to Monitor

(15:31) Backed Into a Corner

(18:41) It's Always RL's Fault

(20:47) Wow Would You Look At Those Goalposts

(22:58) Astra Seems Pretty Misaligned To Me

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.