I took a week off from my day job of aligning AI to visit Forethought and think about the question: if we can align AI, what should we do with it? This post summarizes the state of my thinking at the end of that week. (The proposal described here is my own, and is not in any way endorsed by Forethought.)

Thanks to Mia Taylor, Tom Davidson, Ashwin Acharya, and a whole bunch of other people (mostly at Forethought) for discussion and comments.

And a quick note: after writing this, I was told that Eric Drexler and David Dalrymple were thinking about a very similar idea in 2022, with essentially the same name. My thoughts here are independent of theirs.

The world around the time of ASI will be scary

I expect the time right around when the first ASI gets built to be chaotic, unstable, and scary. [...]

---

Outline:

(00:53) The world around the time of ASI will be scary

(03:39) The night-watchman ASI

(04:57) Three key properties

(05:44) The night watchman's responsibilities

(05:53) First and foremost: keeping the peace

(07:22) Minimally intrusive surveillance

(07:42) Preventing competing ASIs

(08:39) Preserving its own integrity

(08:51) Preventing premature claims to space

(09:54) Preventing other kinds of lock-in

(10:09) Preventing underhanded negotiation tactics

(10:46) Arguing for the three key properties above

(10:59) Getting everyone on board

(12:54) Protecting humanity in the short run

(13:11) No major lock-in

(13:58) Interpretive details

(15:15) Amending the night watchman's goals

(15:56) Modifications to the basic proposal

(16:31) Multiple subsystems

(16:53) An American night watchman and a Chinese night watchman overseeing each other

(17:36) Keeping the peace through soft power

(18:05) Conventional treaties

(18:26) Checks and balances

(18:52) The night watchman as a transition

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

This is a short story I wrote in mid-2022. Genre: cosmic horror as a metaphor for living with a high p-doom.

One

The last time I saw my mom, we met in a coffee shop, like strangers on a first date. I was twenty-one, and I hadn’t seen her since I was thirteen.

She was almost fifty. Her face didn’t show it, but the skin on the backs of her hands did.

“I don’t think we have long,” she said. “Maybe a year. Maybe five. Not ten.”

It says something about San Francisco, that you can casually talk about the end of the world and no one will bat an eye.

Maybe twenty, not fifty, was what she’d said eight years ago. Do the math. Mom had never lied to me. Maybe it would have been better for my childhood if she had [...]

---

Outline:

(04:50) Two

(22:58) Three

(35:33) Four

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

One of our main activities at Redwood is writing follow-ups to previous papers on control like the original and Ctrl-Z, where we construct a setting with a bunch of tasks (e.g. APPS problems) and a notion of safety failure (e.g. backdoors according to our specific definition), then play the adversarial game where we develop protocols and attacks on those protocols.

It turns out that a substantial fraction of the difficulty here is developing the settings, rather than developing the protocols and attacks. I want to explain why this is the case.

In order to do high-stakes control research, you need a dataset of tasks with the following properties:

• You can check whether the main task was completed successfully.
• There's some notion of safety failure that we consider unacceptable; we can check whether a safety failure occurred. (We sometimes call these safety failures "side tasks".)

Here's why it's challenging [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

METR ran a proper RCT experiment seeing how much access to Cursor (using Sonnet 3.7) would accelerate coders working on their own open source repos.

Everyone surveyed expected a substantial speedup. The developers thought they were being substantially sped up.

Instead, it turned out that using Cursor slowed them down.

That surprised everyone, raising the question of why.

Currently our best guess is this comes down to a combination of two factors:

---

Outline:

(01:27) Epic Fail

(02:42) The Core Result

(07:10) Okay So That Happened

(12:21) Beginner Mindset

(19:43) Overall Takeaways

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I am quite harsh in impersonal settings, such as on the Internet or at work. Attention is a scarce resource, and I am stingy with it. The world is Out To Get Us. In many ways, nice or not.

Social Media platforms explicitly try to capture our attention. Children, despite being wonderful, are foremost an attention pit. It's always possible to do more for them. The same goes for People Being Wrong On The Internet or Social Obligations.

At large, too many get got, and do not have enough attention left to reflect. This traps them in situations that they cannot escape: reflecting is what would let them see a way out in the first place.

You ought to meditate 20 minutes a day. Unless you're too busy. Then you ought to meditate 1 hour a day.

Trying the Obvious Thing

This is why I have built some defence [...]

---

Outline:

(01:01) Trying the Obvious Thing

(01:20) The Weak

(02:15) Fake Experts

(04:33) What counts as the Obvious Thing?

(05:25) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

(This is the video and transcript of a public talk I gave at Mox in San Francisco in July 2025, on long-term equilibria post-AGI. It's a longer version of the talk I gave at this workshop. The slides are also available here.)

Introduction

Thank you. Okay. Hi. Thanks for coming.

Aims for this talk

So: can goodness compete? It's a classic question, and it crops up constantly in a certain strand of futurism, so I'm going to try to analyze it and understand it more precisely. And in particular I want to distinguish between a few different variants, some of which are more fundamental problems than others. And then I want to try to hone in on what I see as the hardest version of the problem – and in particular, possible ways good values can have inherent disadvantages and competition with other value systems. For [...]

---

Outline:

(00:30) Introduction

(00:50) Aims for this talk

(01:54) Basic vibe

(03:28) Lineage of concern

(04:31) What I mean by goodness

(07:23) What I mean by competition

(09:27) Can \*humans\* compete?

(11:21) Some distinctions

(14:17) Alignment taxes

(16:52) More fundamental variants

(18:24) Negative sum dynamics

(20:36) The strategy stealing assumption

(21:59) Locust-like value systems

(25:47) Other ways the strategy stealing assumption might fail

(28:43) Addressing failures of the strategy stealing assumption

(32:05) Is preventing/constraining competition in this way even possible?

(34:11) Is preventing/constraining competition in this way desirable?

(36:36) What is a locust world actually like?

(38:03) Might a locust world be less bleak than this?

(44:15) Current overall take

(45:33) Poem: Witchgrass

(47:51) Q&A

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

So I'm "back" on Less Wrong, which is to say that I was surprised to find that I already had an account and had even, apparently, commented on some things. 11 years ago. Which feels like half a lifetime ago. More than half a my-adult-lifetime-so-far. A career change and a whole lot of changes in the world ago.

I've got a funny relationship to this whole community I guess.

I've been 'adj' since forever but I've never been a rat and never, until really quite recently, had much of an interest in the core rat subjects. I'm not even a STEM person, before or after the career change. (I was in creative arts - now it's evidence-based medicine.)

I just reached the one-year anniversery of the person I met on rat-adj tumblr moving accross the ocean for me, for love. So there was always going to be an [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

One of the most underrated essays in the post-Sequences era of Eliezer Yudkowsky's corpus is "Four Layers of Intellectual Conversation". The degree to which this piece of wisdom has fallen into tragic neglect in these dark ages of the 2020s may be related to its ephemeral form of publication: it was originally posted as a status update on Yudkowsky's Facebook account on 20 December 2016 and subsequently mirrored on Alyssa Vance's The Rationalist Conspiracy blog, which has since gone offline. (The first link in this paragraph is to an archive of the Rationalist Conspiracy post.)

In the post, Yudkowsky argues that a structure of intellectual value necessarily requires four layers of conversation: thesis, critique, response, and counter-response (which Yudkowsky indexes from zero as layers 0, 1, 2, and 3).

The importance of critique is already widespread common wisdom: if a thesis is advanced and promulgated without any [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Audio note: this article contains 53 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Ariana Azarbal*, Matthew A. Clarke*, Jorio Cocola*, Cailley Factor*, and Alex Cloud.

*Equal Contribution. This work was produced as part of the SPAR Spring 2025 cohort.

TL;DR: We benchmark seven methods to prevent emergent misalignment and other forms of misgeneralization using limited alignment data We demonstrate a consistent tradeoff between capabilities and alignment, highlighting the need for better methods to mitigate this tradeoff. Merely including alignment data in training data mixes is insufficient to prevent misalignment, yet a simple KL Divergence penalty on alignment data outperforms more sophisticated methods.

Introduction

Training to improve capabilities [...]

---

Outline:

(01:27) Introduction

(03:36) Our Experiments

(04:23) Formalizing the Objective

(05:20) Can we solve the problem just by training on our limited alignment data?

(05:55) Seven methods for selective generalization

(07:06) Plotting the capability-alignment tradeoff

(07:26) Preventing Emergent Misalignment

(10:33) Preventing Sycophantic Generalization from an Underspecified Math Dataset

(14:02) Limitations

(14:40) Takeaways

(15:49) Related Work

(17:00) Acknowledgements

(17:22) Appendix

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I keep meaning to write up a more substantive followup to the Hire (or Become) a Thinking Assistant post. But, I think this still basically the biggest productivity effect size I know of, and more people should be doing it, and seemed worth writing the simple version of the post.

I tried out 4 assistants who messaged me after the previous post, and ultimately settled into a rhythm of using one who had availability that matched my needs best. I think all of them were net helpful.

So far they've all been remote contractors that I share my screen with on zoom. Previously I had worked with one in person which also went well although was a bit harder to schedule with.

Most of what I do is just have them check in on me every 10 minutes and make sure I haven't gotten off track. Sometimes, I [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.