Audio note: this article contains 38 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Paper | Github | Demo Notebook

This post is about our recent paper Learning to Interpret Weight Differences in Language Models (Goel et al. Oct. 2025). We introduce a method for training a LoRA adapter that gives a finetuned model the ability to accurately describe the effects of its finetuning.

WeightDiffQA

Our paper introduces and attempts to solve a task we call WeightDiffQA[1]:

Given a language model _M_, a weight diff _delta_, and a natural language question _q_ about _delta_, output a correct natural language answer to _q_.

Here, a "weight [...]

---

Outline:

(00:57) WeightDiffQA

(03:17) Diff Interpretation Tuning

(05:09) Eval #1: Reporting hidden behaviors

(07:17) Eval #2: Summarizing finetuned knowledge

(08:26) Limitations

(09:50) Takeaways

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It serves people well to mostly build towards a good future rather than getting distracted by the shape of utopia, but having a vision of where we want to go can be helpful for both motivation and as a north star for guiding our efforts.

Publishing prompted by the ACX grant to generate stories of good things with AI.

Opening Vision

Utopiography: What would you like it to be like?

plex: A large proportion of my value structure is pointed at other people's. I would like many different utopias to flourish. I think that the universe is very big and has room for many of the different things that humans could unfold into. And I would like to see it filled with a diverse garden of different mind spaces, different cultures and recombinations and extrapolations of what we had become.

Utopiography: Is your preferred version of that largely [...]

---

Outline:

(00:30) Opening Vision

(02:24) Escaping Moloch

(05:16) The Transition Period

(08:17) Interventions and technology

(12:17) The Jerusalem Problem

(15:56) Space travel

(17:23) Meaning in Paradise

(23:07) Heat Death - The Universes Final Boss

(32:20) Is Eternity Too Much?

(42:53) Life in Utopia

(48:11) Suffering and Choice

(50:51) Population Ethics and Vast but Finite Resources

(56:29) Moral Patients and Suffering

(01:16:40) A Beautiful Tomorrow

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

We call for a prohibition on the development of superintelligence, not lifted before there is

1. broad scientific consensus that it will be done safely and controllably, and
2. strong public buy-in.

Sign on the main website or Eko[1].

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Linkpost URL:
https://superintelligence-statement.org/

---

Narrated by TYPE III AUDIO.

There's an argument I sometimes hear against existential risks, or any other putative change that some are worried about, that goes something like this:

'We've seen time after time that some people will be afraid of any change. They'll say things like "TV will destroy people's ability to read", "coffee shops will destroy the social order","machines will put textile workers out of work". Heck, Socrates argued that books would harm people's ability to memorize things. So many prophets of doom, and yet the world has not only survived, it has thrived. Innovation is a boon. So we should be extremely wary when someone cries out "halt" in response to a new technology, as that path is lined with skulls of would be doomsayers."

Lest you think this is a straw man, Yann Le Cun compared fears about AI doom to fears about coffee. Now, I don't want to criticize [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

It is a strange thing to love another. I had not had much experience. It was Penny, of course, who I fell for. I suppose everyone fell in love with her a little. She is beautiful. Maybe the most beautiful person any of us had ever seen. Even movie stars are not as beautiful as she. They do not put women such as her in films. It is not realistic. Aphrodite could not star in a romantic comedy. Aphrodite has drama in her dealings with gods but not with men.

She looks better, even, than those in fashion magazines. My friend Liam, who is gay, has a theory about it. He claims male models are indeed the epitome of masculine beauty, as you would expect, and he follows their lives on Instagram with great interest - I often mourn for him the sad fact that the traits he most [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

In recent years, I’ve found that people who self-identify as members of the AI safety community have increasingly split into two camps:

Camp A) "Race to superintelligence safely”: People in this group typically argue that "superintelligence is inevitable because of X”, and it's therefore better that their in-group (their company or country) build it first. X is typically some combination of “Capitalism”, “Molloch”, “lack of regulation” and “China”.

Camp B) “Don’t race to superintelligence”: People in this group typically argue that “racing to superintelligence is bad because of Y”. Here Y is typically some combination of “uncontrollable”, “1984”, “disempowerment” and “extinction”.

Whereas the 2023 extinction statement was widely signed by both Camp B and Camp A (including Dario Amodei, Demis Hassabis and Sam Altman), the 2025 superintelligence statement conveniently separates the two groups – for example, I personally offered all US Frontier AI CEO's to sign, and none chose [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Last week I gave an invited talk as part of the Integral Altruism speaker series. A recording of the talk and the extensive Q&A is up on YouTube; it's close to 90 minutes long. If you’d rather read than watch/listen, I had Claude convert the transcript into something more like essay form.

In the talk I try to cover roughly how the postrationalist scene got started and how it later merged with other scenes thinking along similar lines but with different intellectual starting points. If it contains any errors (because you were there and remember it differently than me!), please let me know in the comments.

A Short History of Postrationality

Opening Presentation

Thanks for having me. This is exciting to get to talk about this stuff. I occasionally grab people at conferences and they’ll ask me questions about this or we’ll just get on this topic. It's [...]

---

Outline:

(00:46) A Short History of Postrationality

(00:51) Opening Presentation

(02:27) The Berkeley Origins (2014-2015)

(05:49) The Reading Group and Foundational Texts

(14:31) The Movement Fragments

(18:07) Q&A Section

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Written in my personal capacity. Thanks to many people for conversations and comments. Written in less than 24 hours; sorry for any sloppiness.

It's an uncanny, weird coincidence that the two biggest legislative champions for AI safety in the entire country announced their bids for Congress just two days apart. But here we are.

On Monday, I put out a long blog post making the case for donating to Alex Bores, author of the New York RAISE Act. And today I’m doing the exact same thing for Scott Wiener, who announced a run for Congress in California today (October 22).

Much like with Alex Bores, if you’re potentially interested in donating to Wiener, my suggestion would be to:

1. Read this post to understand the case for donating to Scott Wiener.
2. Understand that political donations are a matter of public record, and that this [...]

---

First published:

Source:

Linkpost URL:
https://ericneyman.wordpress.com/2025/10/22/consider-donating-to-ai-safety-champion-scott-wiener/

---

Narrated by TYPE III AUDIO.

This is the latest in a series of essays on AI Scaling.
You can find the others on my site.

Summary: RL-training for LLMs scales surprisingly poorly. Most of its gains are from allowing LLMs to productively use longer chains of thought, allowing them to think longer about a problem. There is some improvement for a fixed length of answer, but not enough to drive AI progress. Given the scaling up of pre-training compute also stalled, we'll see less AI progress via compute scaling than you might have thought, and more of it will come from inference scaling (which has different effects on the world). That lengthens timelines and affects strategies for AI governance and safety.

The current era of improving AI capabilities using reinforcement learning (from verifiable rewards) involves two key types of scaling:

1. Scaling the amount of compute used for RL during training
2. Scaling [...]

---

Outline:

(09:46) How do these compare to pre-training scaling?

(14:16) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The White House's Office of Science and Technology Policy has issued a request for information (RFI) relevant to regulations that "unnecessarily hinder the development, deployment, and adoption of artificial intelligence (AI) technologies within the United States."

If you have anything to say about that topic that you want the U.S. government to hear, consider submitting a public comment before 11:59 p.m. Eastern Time on Monday 27 October 2025.

The emphasis on removing regulations is pursuant to the White House's AI Action Plan, which was the subject of a previous RFI.

I submitted a one-page commment to that one, which did get dutifully included in the 18,480 page PDF of 10,067 responses. Interestingly, the new RFI on deregulation only has 243 comments at press time for this post; perhaps informative comments are less likely to get drowned out in the noise this time.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.