Critics of Trump often describe him as making absolutely unprecedented moves to expand executive power, extract personal wealth, and impinge on citizens’ rights. Supporters counter that Trump's actions are either completely precedented, or are the natural extension of existing trends that the media wouldn’t make a big deal over if they didn’t hate Trump so much.

In some recent posts, some people have been like "Wait why is there suddenly this abrupt series of partisan LW posts that are taking for granted there is a problem here that is worth violating the LW avoid-most-mainstream-politics norm?".

My subjective experience has been "well, me and most of my rationalist colleagues have spent the past 15 years mostly being pretty a-political, were somewhat wary but uncertain about Trump during his first term, and the new set of incidents just seems... pretty unprecedently and scarily bad?"

But, I do definitely live in a bubble that serves me tons of news about bad-seeming things that Trump is doing. It's possible to serve up dozens or hundreds of examples of a scary thing per day, without that thing actually being objectively scary or abnormal. (See: Cardiologists and Chinese Robbers)

Elizabeth and I wanted to [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

The recent book “If Anyone Builds It Everyone Dies” (September 2025) by Eliezer Yudkowsky and Nate Soares argues that creating superintelligent AI in the near future would almost certainly cause human extinction:

If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.

The goal of this post is to summarize and evaluate the book's key arguments and the main counterarguments critics have made against them.

Although several other book reviews have already been written I found many of them unsatisfying because a lot of them are written by journalists who have the goal of writing an entertaining piece and only lightly cover the core arguments, or don’t seem understand them properly, and instead resort to weak arguments like straw-manning, ad hominem attacks or criticizing the style of the book.

So my goal is to write a book review that has the following properties:

• Written by someone who has read a substantial amount of AI alignment and LessWrong content and won’t make AI alignment beginner mistakes or misunderstandings (e.g. not knowing about the [...]

---

Outline:

(07:43) Background arguments to the key claim

(09:21) The key claim: ASI alignment is extremely difficult to solve

(12:52) 1. Human values are a very specific, fragile, and tiny space of all possible goals

(15:25) 2. Current methods used to train goals into AIs are imprecise and unreliable

(16:42) The inner alignment problem

(17:25) Inner alignment introduction

(19:03) Inner misalignment evolution analogy

(21:03) Real examples of inner misalignment

(22:23) Inner misalignment explanation

(25:05) ASI misalignment example

(27:40) 3. The ASI alignment problem is hard because it has the properties of hard engineering challenges

(28:10) Space probes

(29:09) Nuclear reactors

(30:18) Computer security

(30:35) Counterarguments to the book

(30:46) Arguments that the books arguments are unfalsifiable

(33:19) Arguments against the evolution analogy

(37:38) Arguments against counting arguments

(40:16) Arguments based on the aligned behavior of modern LLMs

(43:16) Arguments against engineering analogies to AI alignment

(45:05) Three counterarguments to the books three core arguments

(46:43) Conclusion

(49:23) Appendix

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I'm glad I know this, and maybe some people here don't, so here goes.

You might also want to expand the denominator using the law of total probability, since you're more likely to know how probable the evidence is given different hypotheses than in general:

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

(This post elaborates on a few ideas from my review of Sam Eisenstat's Condensation: a theory of concepts. It should be somewhat readable on its own but doesn't fully explain what condensation is on its own; for that, see my review or Sam's paper. The post came out of conversations with Sam.)

As I mentioned in my Condensation review, the difference between compression and condensation fits the physical analogy suggested by their names: compression mashes all the information together, while condensation (still compresses size, but) sorts information into discrete droplets.

Thus, condensation has a property we might call local relevance: typical questions can be answered at a glance, ie, retrieving small subsets of the information. This type of representation is sometimes called "symbolic":

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Anthropic released a new constitution for Claude. I encourage those interested to read the document, either in whole or in part. I intend to cover it on its own soon.

There was also actual talk about coordinating on a conditional pause or slowdown from both DeepMind CEO Demis Hassabis and Anthropic CEO Dario Amodei, which I also plan to cover later.

Claude Code continues to be the talk of the town, the weekly report on that is here.

OpenAI responded by planning ads for the cheap and free versions of ChatGPT.

There was also a fun but meaningful incident involving ChatGPT Self Portraits.

Table of Contents

---

Outline:

(00:52) Language Models Offer Mundane Utility

(01:25) Language Models Don't Offer Mundane Utility

(01:55) Huh, Upgrades

(02:48) Gemini Personalized Intelligence

(07:06) Deepfaketown and Botpocalypse Soon

(10:34) Fun With Media Generation

(11:35) We're Proud To Announce The Torment Nexus

(18:58) They Took Our Jobs

(25:29) The Revolution of Rising Expectations

(26:30) Get Involved

(27:52) A Young Lady's Illustrated Primer

(30:30) In Other AI News

(32:57) Axis of Assistance

(39:58) Show Me the Money

(40:39) California In Crisis

(42:14) Bubble, Bubble, Toil and Trouble

(43:57) Quiet Speculations

(52:32) Elon Musk Versus OpenAI

(56:31) The Quest for Sane Regulations

(01:03:30) Chip City

(01:08:57) The Week in Audio

(01:14:19) Rhetorical Innovation

(01:19:47) Aligning a Smarter Than Human Intelligence is Difficult

(01:23:06) Alignment Is Not Primarily About a Metric

(01:30:43) How To Be a Safe Robot

(01:37:52) Living In China

(01:41:05) Claude 3 Opus Lives

(01:41:23) People Are Worried About AI Killing Everyone

(01:42:42) Messages From Janusworld

(01:43:53) Everyone Is Confused About AI Consciousness

(01:45:36) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Last June, METR caught o3 reward hacking on its RE-Bench and HCAST benchmarks. In a particularly humorous case, o3, when tasked with optimizing a kernel, decided to “shrink the notion of time as seen by the scorer”.

The development of Humanity's Last Exam involved “over 1,000 subject-matter experts” and $500,000 in prizes. However, after its release, researchers at FutureHouse discovered “about 30% of chemistry/biology answers are likely wrong”.

LiveCodeBench Pro is a competitive programming benchmark developed by “a group of medalists in international algorithmic contests”. Their paper describes issues with the benchmark's predecessor:

Benchmarks like LiveCodeBench [35] offer coding problems, but suffer from inconsistent environments, weak test cases vulnerable to false positives, unbalanced difficulty distributions, and the inability to isolate the effects of search contamination.

However, the authors assure us that their own test cases are of high quality:

Many problems in our benchmark originate from Codeforces, which uses the Polygon problem-setting platform. Each problem is then rigorously vetted by a team of expert testers—typically drawn from the community's top 1%, and overseen by at least one coordinator, usually among the top 0.1%. These specialists verify both the soundness and originality of every problem, ensuring it has never appeared [...]

---

Outline:

(02:38) Terminal-Bench 2 Audit

(05:59) Why does this matter?

(07:15) What to do about it

(09:22) Appendix: More benchmark issues

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TLDR: Will AI-automation first speed up capabilities or safety research? I forecast that most areas of capabilities research will see a 10x speedup before safety research. This is primarily because capabilities research has clearer feedback signals and relies more on engineering than on novel insights. To change this, researchers should now build and adopt tools to automate AI Safety research, focus on creating benchmarks, model organisms, and research proposals, and companies should grant differential access to safety research.

Epistemic status: I spent ~ a week thinking about this. My conclusions rely on a model with high uncertainty, so please take them lightly. I’d love for people to share their own estimates about this.

The Ordering of Automation Matters

AI might automate AI R&D in the next decade and thus lead to large increases in AI progress. This is extremely important for both the risks (eg an Intelligence Explosion) and the solutions (automated alignment research).

On the one hand, AI could drive AI capabilities progress. This is a stated goal of Frontier AI Companies (eg OpenAI aims for a true automated AI researcher by March of 2028), and it is central to the AI 2027 forecast. On the other hand [...]

---

Outline:

(01:00) The Ordering of Automation Matters

(05:39) Methodology

(11:43) Prediction

(13:42) Levers for affecting the order

(14:35) Speeding up Safety Automation

(19:09) Future Work: How could this be researched properly?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

If we get AI safety research wrong, we may not get a second chance. But despite the stakes being so high, there has been no effort to systematically review and verify empirical AI safety papers. I would like to change that.

Today I sent in funding applications to found a team of researchers dedicated to replicating AI safety work. But what exactly should we aim to accomplish? What should AI safety replications even look like? After 1-2 months of consideration and 50+ hours of conversation, this document outlines principles that will guide our future team.

I. Meta-science doesn’t vindicate anyone

Researchers appear to agree that some share of AI safety work is low-quality, false, or misleading. However, everyone seems to disagree on which share of papers are the problematic ones.

When I expressed interest in starting a group that does AI safety replications, I suspect some assumed I would be “exposing” the papers that they don’t approve of. This is a trap and it is especially important for us, as the replicators, not to fall into it. If our replications tend to confirm our beliefs, that probably says more about our priors than the papers we are studying.

[...]

---

Outline:

(00:48) I. Meta-science doesn't vindicate anyone

(01:27) II. Searching for bad papers is like searching for haunted houses

(02:22) III. Research doesn't regulate itself

(03:29) IV. Replications are more than repeating the experiments

(04:23) V. The replication is just as dubious as the paper itself

(05:11) VI. Why not do this in a more decentralized way?

(06:07) VII. We are all adults here

(06:46) VIII. Feedback is everything

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

A new version of “Intro to Brain-Like-AGI Safety” is out!

Things that have not changed

Same links as before:

• LessWrong / Alignment Forum blog version: https://www.lesswrong.com/s/HzcM2dkCq7fwXBej8
• Archival PDF version (now version 3!): https://osf.io/preprints/osf/fe36n
• Summary video: Video & transcript: Challenges for Safe & Beneficial Brain-Like AGI

…And same abstract as before:

Suppose we someday build an Artificial General Intelligence algorithm using similar principles of learning and cognition as the human brain. How would we use such an algorithm safely?

I will argue that this is an open technical problem, and my goal in this post series is to bring readers with no prior knowledge all the way up to the front-line of unsolved problems as I see them.

Post #1 contains definitions, background, and motivation. Then Posts #2–#7 are the neuroscience, arguing for a picture of he brain that combines large-scale learning algorithms (e.g. in the cortex) and specific evolved reflexes (e.g. in the hypothalamus and brainstem). Posts #8–#15 apply those neuroscience ideas directly to AGI safety, ending with a list of open questions and advice for getting involved in the field.

A major theme will be that the human brain runs a yet-to-be-invented variation on Model-Based Reinforcement Learning. [...]

---

Outline:

(00:16) Things that have not changed

(02:40) Highlights from the changelog

(02:57) Post 1: Whats the problem & Why work on it now?

(03:03) What is AGI?

(05:49) More responses to intelligence denialists

(08:27) Post 2: Learning from scratch in the brain

(08:32) Better overview of the discourse

(12:05) Plasticity

(14:20) Interpretability

(16:18) Post 3: Two subsystems: Learning & Steering

(16:24) My timelines prediction

(17:24) Responses to bad takes on acting under uncertainty

(19:06) Post 5. The long-term predictor, and TD learning

(19:13) More pedagogy on the toy model

(20:11) Post 6: Big picture of motivation, decision-making, and RL

(20:18) More on why ego-syntonic goals are in the hypothalamus & brainstem

(22:45) Post 10: The technical alignment problem

(22:51) LLMs

(23:09) 10.3.1.1. Didn't LLMs solve the Goodhart's Law problem?

(25:58) Instrumental convergence & consequentialist preferences

(26:14) 10.3.2.3. Motivations that don't lead to instrumental convergence

(28:18) What about RL today?

(31:24) What do I mean by (technical) alignment?

(34:34) Post 12: Two paths forward: Controlled AGI and Social-instinct AGI

(34:43) What exactly is the RL training environment?

(37:08) Post 15: Conclusion: Open problems, how to help, AMA

(37:15) Reward Function Design

(38:59) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

As soon as modern data analysis became a thing, the US government has had to deal with people trying to use open source data to uncover its secrets.

During the early Cold War days and America's hydrogen bomb testing, there was an enormous amount of speculation about how the bombs actually worked. All nuclear technology involves refinement and purification of large amounts of raw substances into chemically pure substances. Armen Alchian was an economist working at RAND and reasoned that any US company working in such raw materials and supplying the government would have made a killing leading up to the tests.

After checking financial data that RAND maintained on such companies, Alchian deduced that the secret sauce in the early fusion bombs was lithium and the Lithium Corporation of America was supplying the USG. The company's stock had skyrocketed leading up to the Castle Bravo test either by way of enormous unexpected revenue gains from government contracts, or more amusingly, maybe by government insiders buying up the stock trying to make a mushroom-cloud-sized fortune with the knowledge that lithium was the key ingredient.

When word of this work got out, this story naturally ends with the FBI coming [...]

---

Outline:

(01:27) Pizza is the new lithium

(03:09) The Data

(04:11) The Backtest

(04:36) Fordow bombing

(04:55) Maduro capture

(05:15) The Houthi stuff

(10:25) Coda

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.