I worked a bunch on the website for If Anyone Builds Its Online Resources. It went through a lot of revisions in the weeks before launch.

There was a particular paragraphs I found important, which I now can't find a link to, and I'm not sure if they got deleted in an edit pass or if they just moved around somewhere I'm failing to search for.

It came after a discussion of corrigibility, and how MIRI made a pretty concerted attempt at solving it, which involved bringing in some quite smart people and talking to people who thought it was obviously "not that hard" to specify a corrigible mind in a toy environment.

The paragraph went (something like, paraphrased from memory):

The technical intuitions we gained from this process, is the real reason for our particularly strong confidence in this problem being hard."

This seemed like a pretty [...]

---

Outline:

(03:21) Intelligent (Usually) Implies Incorrigible

(10:42) Shutdown Buttons and Corrigibility

(23:42) A Closer Look at Before and After

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

California Governor Gavin Newsom signed SB 53 on September 29. I think it's a pretty great step, though I certainly hope future legislation builds on it.

I wrote up my understanding of what the text actually does; I welcome any corrections (and certainly further analysis for what this all means for AI safety)!

Very short summary

The law requires major AI companies to:

• Publish a frontier AI framework describing how the developer “approaches” assessing and mitigating catastrophic risks, defined as risks of death/serious injury to >50 people or >$1B in damage from a single incident related to CBRN uplift, autonomous crime, or loss of control, and keep the published version of the framework up to date.
• Publish model cards summarizing the assessments of catastrophic risks from the model, the role of third parties in those assessments, and how the frontier AI framework was implemented.
• Report to California's Office of Emergency [...]

---

Outline:

(00:32) Very short summary

(02:22) Longer summary

(02:25) What the bill requires of large frontier developers

(07:38) Other notes about the bill

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

This seems like a good opportunity to do some of my classic detailed podcast coverage.

The conventions are:

Full transcript of the episode is here if you want to verify exactly what was said.

Well, that was the plan. This turned largely into me quoting Sutton and then expressing my mind boggling. A lot of what was interesting [...]

---

Outline:

(01:21) Sutton Says LLMs Are Not Intelligent And Don't Do Anything

(13:51) Humans Do Imitation Learning

(19:35) The Experimental Paradigm

(23:45) Current Architectures Generalize Poorly Out Of Distribution

(28:36) Surprises In The AI Field

(30:01) Will The Bitter Lesson Apply After AGI?

(34:18) Succession To AI

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I

The popular conception of Dunning-Kruger is something along the lines of “some people are too dumb to know they’re dumb, and end up thinking they’re smarter than smart people”. This version is popularized in endless articles and videos, as well as in graphs like the one below.

Except that's wrong.

II

The canonical Dunning-Kruger graph looks like this:

Notice that all the dots are in the right order: being bad at something doesn’t make you think you’re good at it, and at worst damages your ability to notice exactly how incompetent you are. The actual findings of professors Dunning and Kruger are more consistent with “people are biased to think they’re moderately above-average, and update away from that bias based on their competence or lack thereof, but they don’t [...]

---

Outline:

(00:12) I

(00:39) II

(01:32) III

(04:22) IV

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Strong regulation is not on the table and all US frontier AI companies oppose it to varying degrees. Weak safety-relevant regulation is happening; some companies say they support and some say they oppose. (Some regulation not relevant to AI safety, often confused, is also happening in the states, and I assume companies oppose it but I don't really pay attention to this.) Companies besides Anthropic support federal preemption of state laws, even without an adequate federal framework. Companies advocate for non-regulation-y things like building government capacity and sometimes export controls.

My independent impression is that current regulatory efforts—SB 53 and the RAISE Act—are too weak to move the needle nontrivially. But the experts I trust are more optimistic, and so all-things-considered I think SB 53 and the RAISE Act would be substantially good (for reasons I don't really understand).

This post is based on my resource https://ailabwatch.org/resources/company-advocacy; see that [...]

---

Outline:

(01:23) US state bills

(01:27) SB 1047 (2024)

(02:15) SB 53 (2025)

(02:38) RAISE Act (2025)

(03:05) US federal preemption

(03:41) EU AI Act

(04:22) Super PACs

(05:01) Policies companies support

(05:31) Misc

(05:34) Anthropic & Clark

(06:05) OpenAI & Lehane

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I sometimes joke darkly on Discord that the machines will grant us a nice retirement tweaked out on meth working in a robotics factory 18 hours a day. I don't actually believe this because I suspect an indifferent AGI will be able to quickly replace industrial civilization; it will just kill us and eat the world with nanobots. But many don't believe that Drexlerian nanotechnology is possible.

I am an idiot with little physics knowledge, so possibly this is only interesting from an amplification via debate frame, but this is why I think something close to Drexlerian nanotechnology (or having similar practical implications to Drexlerian nanotechnology) is likely possible:

The design space is very large. Any proof that Drexlerian nanotechnology is impossible would have to be very robust. And it strikes me that most plausible looking arguments can likely be hacked around given how large the design space [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I was slightly worried about sociohazards of posting this here, but the cat's already out of the bag on this one: the authors of the paper put it up on BioRXiv and got an article written about them in Nature, as well as getting into a bunch of newspapers and all over twitter. I expect that the marginal audience of LessWrong is much more heavily enriched in people who'll use this information to do good than to do evil.

TL;DR:
If anyone reading this works in Biosecurity: nucleotide synthesis companies should start using protein structure annotation as part of their screening protocol for genes, if they aren't already.

Intro

The Fermi Paradox is the question of why we don't see any evidence of aliens spreading out across the galaxy, despite there being loads of planets on which intelligent life could evolve.

The Germy Paradox is the question of [...]

---

Outline:

(00:48) Intro

(01:23) I

(04:34) II

(06:52) III

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

9/11 and the immediate ractions to it has one of the biggest distortion fields of any topic in the English language. There is a polite fiction maintained when any Americans are present that everyone everywhere was sad about the incident. We shake our head, frown, and say that everyone shed a tear when we saw the towers fall on TV.

The truth is, there were celebrations around the world. I have talked to multiple Chinese people who were in the US at the time who told me some variant of "I was absolutely celebrating on the inside, and honestly kinda pissed that I had to pretend to be sad publicly". I had a middle school teacher in China who told a story about how he was literally the only people in a room who wasn't publically celebrating people burning to death in the towers. Something about the NATO bombing [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

An extended version of this article was given as my keynote speech at the 2025 LessWrong Community Weekend in Berlin.

A couple of years ago, I agreed to give a talk on the topic of psychology. I said yes, which of course meant that I now had a problem.

Namely, I had promised to give a talk, but I did not have one prepared. (You could also more optimistically call this an opportunity, a challenge, a quest, etc.)

So I decided to sit down at my computer, work on my presentation, and then go from the world where I had no talk prepared to a world where I did have a talk prepared.

However, I found that it did not quite work out that way. Whenever I tried moving toward the goal of having a finished talk, I instead found myself moving sideways.

And whenever I did end up [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Let's say you’re a zoo architect tasked with designing an enclosure for ostriches, and let's also say that you have no idea what an ostrich is (roll with me here). The potentially six-figure question staring you down is whether to install a ceiling.

The dumb solution is to ask “Are ostriches birds?” then, surmising that birds typically fly, construct an elaborate aviary complete with netting and elevated perches.

The non-dumb solution is to instead ask “Can ostriches fly?” then skip the ceiling entirely, give these 320-pound land-bound speed demons the open sky life they actually need, and pocket the cost differential.

Your boss is not having it, however. When you inform him of your decision to eschew the ceiling enclosure, he gets beet-red and apoplectic, repeating like a mantra “But they’re birds! BIRDS!” and insists that ostriches belong in the aviary with the finches and parrots. I mean, he's [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.