The context for this post is primarily Only Law Can Prevent Extinction, but after first drafting a half-assed comment, I decided to get off my ass and write a whole-assed post.

I agree with Eliezer's main thesis that individual violence against AI researchers is both morally wrong and strategically stupid. Where I disagree is with the claim that international law can prevent extinction. It can't, for the following reasons.

I. International law is largely a fiction (especially when interests diverge sharply)

The analogy with nuclear weapons is a poor one. North Korea signed the nuclear non-proliferation treaty and developed nuclear weapons anyway. The treaty deterred only those who weren't very motivated anyway. And the reason why the US and Russia didn't nuke each other has nothing to do with international treaties (see point II).

In practice, powerful countries disregard international law whenever they want. A stark example of this is the Budapest Memorandum: in 1994, Ukraine surrendered all its nuclear warheads in exchange for written sovereignty guarantees from Russia, the US, and the UK. Russia annexed a part of Ukraine in 2014, and the international community expressed concern. Russia launched a full-scale invasion in 2022, and the first thing [...]

---

Outline:

(00:36) I. International law is largely a fiction (especially when interests diverge sharply)

(02:01) II. The AI race is perceived as asymmetrical, unlike nuclear MAD

(03:03) III. There is virtually zero possibility of consensus on AI risk, unlike nuclear weapons

(04:41) IV. The proposed enforcers have a demonstrated track record of not enforcing things

(05:30) V. GPU control is not analogous to nuclear material control

(06:58) VI. A flawed treaty is not better than nothing

(08:26) So is there a better way?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Overview:

We train a tiny ReLU network to output sparse top- distributions over a vocabulary much larger than its residual dimension. The trained network seems to converge to a mechanism closely resembling a Bloom filter: tokens are assigned sparse binary hashes, the hidden layer computes an approximate union indicator, and the output logits are linearly read from this union.

Here's what a small network trained on a toy version of the sparse top- distribution task learns to use:

Weight matrix of a 1-layer ReLU network trained via gradient descent on the toy -sparse distribution task below, for , , . Truncated at first tokens for visualisation purposes.

Plot of the range of values of , it forms a bimodal distribution.

That's the input weight matrix of the trained network. Every entry is either or . The network has effectively encoded a binary hash for each token - and as we'll show, this seems to enable the network to approximately simulate a Bloom filter, and so output the correct set of top- tokens with high probability.

We provide a theoretical construction showing how to set the weights to exactly implement a Bloom filter. The real network [...]

---

Outline:

(00:10) Overview:

(02:02) The Task:

(03:27) Construction:

(04:17) Formal construction:

(04:47) Analysis of a single forward pass:

(06:13) Training:

(07:04) Behavioural analysis of the trained network:

(10:14) Mechanistic analysis of the trained network:

(16:21) Conclusion / Reflections:

(18:24) Related work:

(19:25) Further work:

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I should introduce myself briefly. I'm an independent researcher, striving to understand human consciousness. My research is available at smoothbrains.net. I often work in loose collaboration with a nonprofit called the Qualia Research Institute. We hope to use human phenomenology to inform the construction of structural models of subjective experience – both to help evaluate the viability of different theories of consciousness, and to better model the welfare of sentient beings.

Contemporary debate over the moral patienthood of digital minds misses the forest for the trees. Mainstream opinion is divided into physicalist and computationalist camps, who believe that consciousness is substrate dependent and substrate independent, respectively. For this reason, those on the physicalist side frequently make the claim that digital computers will never be conscious. Personally, I consider myself a physicalist, but I'm also a panpsychist – because physics doesn't really seem to deal in hard absolutes, and I find it straightforward to consider that everything is conscious to some greater or lesser degree – so I'm loath to accept any claims which propose that any specific system isn't conscious. I think statements such as these are not defensible, and only serve to encourage misunderstanding and even foment philosophical [...]

---

Outline:

(04:17) My position statement

(08:50) My argument

(09:07) 1. The translation problem

(10:26) Building a physicalist translation function

(19:39) 2. The simplicity problem

(20:31) Computationalist translation functions are observer dependent

(25:17) 3. The introspection problem

(27:44) Digital hardware prohibits phenomenal introspection

(29:32) Conclusion

(32:56) My research

(33:33) 1. Is the brain an optical computer?

(34:16) 2. If the brain is an optical computer, how is it constructed?

(34:59) 3. How do we ensure the well-being of conscious computers?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Availability Problem:

Imagine you have cancer, or chronic pain, or a progressive degenerative disease of some sort. You have exhausted the traditional treatment options available to you, and none of them have worked. However, there are treatments that are still undergoing clinical trials which might help you. They are not fully approved yet, but your situation is dire and you don’t have time to wait another 10 years for the trials to finish. Can you access those treatments?

In theory yes, you can access unapproved treatments through federal laws like the 2018 Right to Try bill, or through FDA pathways like “Expanded Access”. However these laws don’t mandate that the company making the drug gives it to you. And what you will find when you try to use your Right to Try, or Expanded Access, is that there are almost no treatments available for use.

That's why despite there being somewhere in the neighborhood of 13,000,000 Americans with terminal or serious illness, the FDA only grants about 2,000 Expanded Access requests per year, even though they approve 99% of all requests, typically within 24 hours. There just aren’t enough companies even bothering to apply.

No one [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

When I started this series, everyone was going crazy for coding agents.

Now a lot more people are going crazy for coding agents, as well they should given how much better coding agents keep getting, but also Everybody Knows they are good and is focusing on actually using them. With the slower pace of news here it's no longer clear that the waits associated with doing these updates on their own are worthwhile, so I’m going to fold these updates into the weekly again for now unless there's a new major development.

Table of Contents

Whoops, Sorry

Claude Code suffered in April from three distinct issues that have now been fixed.

---

Outline:

(00:38) Whoops, Sorry

(01:45) Huh, Upgrades

(04:13) Codex of Ultimate Computer Use

(08:05) Rookie Numbers

(09:12) I See What You Did There

(11:38) Just a Ride

(11:50) They Didnt Want Our Jobs

(18:30) Skilling Up

(22:08) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The simple mental picture

A simple mental picture we have for an AI capability benchmark is to think of it as a sensor with a certain sensitivity within a certain range of capabilities. The sensitivity of a benchmark, i.e. it's ability to distinguish the capability of different models, is given by a curve like this:

The curve starts high (low sensitivity, high uncertainty), since for models with low capability all the tasks in the benchmark are too hard, and the benchmark can't distinguish between low and very low capability. Similarly all the tasks are too easy for a very capable model, and we lose the ability to differentiate again. In between is the range of capabilities the benchmark is sensitive to, and the sensitivity curve tells you how easy it is to distinguish small capability differences between models at different overall capability levels.

A good benchmark is very sensitive over a long range of capabilities, but there is a tradeoff. Say you want to make a benchmark with 1000 questions. You could make the questions all have roughly the same difficulty. That would make you very sensitive to capabilities close to that difficulty, but you would only [...]

---

Outline:

(00:09) The simple mental picture

(02:06) Epoch Capability Index (ECI) as a toy model

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Here's a dynamic I’ve seen at least a dozen times:

Alice: Man that article has a very inaccurate/misleading/horrifying headline.

Bob: Did you know, *actually* article writers don't write their own headlines?

…

But what I care about is the misleading headline, not your org chart

__

Another example I’ve encountered recently is (anonymizing) when a friend complained about a prosaic safety problem at a major AI company that went unfixed for multiple months. Someone else with background information “usefully” chimed in with a long explanation of organizational limitations and why the team responsible for fixing the problem had limitations on resources like senior employees and compute, and actually not fixing the problem was the correct priority for them etc etc etc.

But what I (and my friend) cared about was the prosaic safety problem not being fixed! And what this says about the company's ability to proactively respond to and fix future problems. We’re complaining about your company overall. Your internal team management was never a serious concern for us to begin with!

__

A third example comes from Kelsey Piper.

Kelsey wrote about the (horrifying) recent case where Hantavirus carriers in the recent [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

The ones who are most successful at writeathons (Inkhaven, NaNoWriMo) are those with an overhang of things to say, usually in the form of:

• draft posts
• daydreams

When Scott Alexander said:

"Whenever I see a new person who blogs every day, it's very rare that that never goes anywhere or they don't get good. That's like my best leading indicator for who's going to be a good blogger." (source).

, it may seem you can just write every day, but that'd be Goodharting. There's something hidden in the writing process you can't see: they have something to say.

They'll have an idea (somehow) and think it through by [writing it out/sitting quietly/etc]. This can then generate more ideas, some of which aren't even related to the original idea!

At this point, though, my imaginary interlocutor would like to say:

I'm trying to publish a blog post every day, so of course I'll eventually be bottlenecked on ideas! How do you generate them though?

Catching Ideas

Have an idea? Write down the idea. This is equivalent to giving your idea-generating process a cookie, reinforcing the habit of generating ideas.

Sometimes, when I'm writing one post, a different idea will [...]

---

Outline:

(01:18) Catching Ideas

(02:32) Just \[Write\] and Nobody Will Get Hurt

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

ProgramBench is a new coding benchmark that all frontier models fail spectacularly. We’ve been on a quest for “hard benchmarks” for a while so it's refreshing to see a benchmark where top models do badly. Unfortunately, ProgramBench has one big problem: it's impossible!

What is ProgramBench?

ProgramBench tests if a model can recreate a program from a “clean room” environment. The model is given only a bit of documentation and black-box access to the program (all the programs are CLIs), then tasked with re-implementing it.

How does ProgramBench know if the implementation is correct? It also generates a bunch of unit tests for the program[1]. The re-implementing coding agent doesn't have access to any of those tests. The coding agent only considers a task “resolved” if it passes all of the tests and “almost resolved” if it passes 95% of them.

Why is this problematic?
Obscure behavior can enter the unit tests without being in the clean room path. An extreme version of this is a backdoor: program that behaves in one way most of the time but behaves totally differently when exposed to a specific string. This wouldn't make a task literally impossible, just incredibly hard in [...]

---

Outline:

(00:37) What is ProgramBench?

(02:41) This seems like a theoretical issue, does it actually happen?

(03:11) What can we do differently?

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.