TL;DR:

Bun is a very large and very influential open-source project. It is being migrated from the easier-to-read Zig programming language to harder-to-read but memory-safe Rust. This is done almost entirely by the AI tool Claude Code. The migration may become one of the earliest major case studies of human control over a significant software project becoming increasingly indirect--both due to the nature of the programming language itself and LLM-generated code being quite convoluted.

What happened?

On May 14 2026, a Rust version of Bun was merged to the "main" branch of the Bun repository[1].

Bun was originally written in Zig, by humans. Then they started to use AI coding tools, likely with an increased intensity after their acquisition by Anthropic in December 2025[2].

Jared Sumner, Bun's creator, posted that the Zig version will be discontinued in favor of the Rust version.

Why is this a big deal?

As far as I'm aware, this is the first example of a hand-written large open-source software project entirely transitioning to LLM-written code.

According to what Jarred Sumner says on X, they haven't been typing code themselves, even before the acquisition--in other words, Claude was already Bun's de facto primary coder.

[...]

---

Outline:

(00:12) TL;DR:

(00:44) What happened?

(01:28) Why is this a big deal?

(02:26) An explosion of complexity

(03:56) Why should you care?

(04:01) A great gamble

(05:26) How gradual disempowerment comes into the picture

(06:35) Appendix: Why did this happen?

(06:56) Memory Safety and Debugging Cost

(08:57) "AI-native" Development Workflow

(09:54) Zig's Anti-LLM Policy

(10:13) Anthropic's Influence

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In philosophy of mind, "mental causation" means mental entities have causal effects, especially physical ones. If physicalism is true, then physical effects are explainable in terms of physical causes (or at least, fundamental physical laws), needing no recourse to causation by anything that is not in fundamental physics. This is the "causal exclusion principle" explicated by Jaegwon Kim (and recently cited in "The Abstraction Fallacy..."), which suggests that, if physicalism is true, then mental entities cannot causally affect anything physical, except insofar as they are already physical entities.

Substance dualists believe in mental causation rather straightforwardly: they believe that the soul has physical effects. Of course, substance dualism contradicts standard physics and physicalism. Type-identity physicalists believe that mental kinds reduce to physical kinds, and that as such, mental causation is a form of physical causation. Mental causation is contrasted with epiphenomenalism, a view under which physical causes can have mental effects but not vice versa.

Epiphenomenalism (e.g. in property dualist form) faces a number of epistemic problems:

---

Outline:

(03:43) Epiphenomenalist functionalism

(07:14) Russellian monism

(11:44) Type-identity physicalism

(15:06) Structural realism and causation

(18:24) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Instead of using static position increments (+1) per token, RoPE-based language models can learn per-token and per-layer position increments. This has no detectable effect on model performance but allows us to see what the model thinks the distance is between each position and how this varies per-layer.

Example sentence with each character plotted based on per-layer learned position increments. Note the clear punctuation-based boundaries in L0 and what looks like concept-based grouping in L3.

I think this might be useful as another technique to inspect "where the model is looking" in addition to plotting attention patterns (and with similar limitations). The patterns can also hint at what the model is looking for at each layer (when position increments match different kinds of boundaries).

Note: This is still partially a solution in search of a problem. I'm hoping to help with the "searching under lamp posts" problem by finding more lamp posts, but there's additional work to be done here to see if this is actually useful or just a novelty.

AI disclaimer: The Architecture, Learned Position Increments, and Related Work sections were originally drafted by Claude before being (heavily) human-edited.

Introduction

Standard LLMs use Rotary Position Embeddings (RoPE) to [...]

---

Outline:

(01:20) Introduction

(01:52) Method

(01:55) Architecture

(02:43) Learned position increments

(04:22) Data and training

(05:20) Results

(05:23) Per-Token Increments

(06:44) First Layer of Per-Layer Model

(07:17) Chinese Word Boundaries

(08:30) Per-Layer Plots

(09:16) Grouping Multi-word Entities

(10:27) Loss Neutral

(11:11) Limitations

(11:49) Future Work

(14:14) Related Work

(15:34) Code

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We introduce an evaluation for activation verbalizers: can they surface a target model's reasoning as it solves a math problem in a single forward pass? For open-weight NLAs, the answer seems to be: "possibly, but definitely not reliably".

Lots of important capabilities currently require AI models to reason "out loud" in a natural-language chain of thought, which means that we can monitor important parts of their thinking. It would be nice to have this same affordance for the reasoning that models do within a single forward pass, especially if the sophistication of that opaque reasoning increases to potentially dangerous levels.

Some interpretability tools might offer such an affordance. In particular, an activation verbalizer (AV) takes a residual stream activation and maps it to a natural-language verbalization. An AV is initialized from the target model and trained to generate verbalizations that an activation reconstructor (AR), also initialized from the target model, can accurately map back to the original activation. Together, an AV and its AR form a natural-language autoencoder (NLA). Importantly, AVs see only a single activation; they do not see the target model's prompt or next-token output, and – unlike activation oracles (AOs) – they are not asked any [...]

---

Outline:

(02:32) Takeaways

(02:43) These NLAs aren't that great at reconstruction

(05:20) It's not clear that NLAs aren't just confabulating from their best guess at the target model's prompt and output

(07:22) Verbalizations do not reliably predict the model's output

(10:03) Verbalizations can surface some hints of reasoning, but rarely with any coherence

(13:02) Verbalizations do not reliably surface causes of wrong outputs

(15:02) More speculative notes

(20:56) Evaluating verbalizations on their ability to recover mathematical reasoning

(24:35) Outputs & Answers - often mentioned, rarely singled out

(27:16) Methods, Values, & Coherence - some early indications

(30:07) Mistakes - almost no plausible explanations of wrong outputs

(33:22) Problems - low-fidelity reconstructions

(34:54) Appendix A: Sample verbalizations

(35:12) Problem 301 (comparison)

(35:33) Gemma

(36:20) Llama

(37:12) Qwen3 AO

(37:56) Qwen2.5

(38:39) Takeaways

(38:42) Problem 533.

(40:48) Problem 448.

(41:55) Problem 549.

(44:31) Problem 691.

(45:46) Problem 340.

(46:47) Output & Answers

(46:50) Problem 824.

(47:46) Problem 479.

(48:53) Problem 509

(49:54) Methods, Values, & Coherence

(49:59) Problem 609

(52:47) Problem 423.

(54:22) Problem 456.

(56:36) Problem 341.

(58:24) Problems

(58:27) Problem 886.

(59:34) Appendix B: Grader elicitation

(01:05:26) Appendix C: Red-teaming

(01:10:42) Appendix D: Steering on mistakes

(01:12:24) Appendix E: Eliciting verbalizations from a question-answering activation oracle

(01:14:24) Problem 545 - Qwen3 AO (confabulation)

(01:15:20) Problem 47 - Qwen3 AO (apparently-good CoT)

(01:16:28) Problem 216 - Qwen3 AO (grader-tricking confabulation)

(01:17:44) Appendix F: Full-dataset results

(01:18:55) Appendix G: cross-layer results

The original text contained 18 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Epistemic status: don’t know whether I actually believe all of this, but I think it's worth considering.

A “corrigible” agent, per the LW wiki, is:

…one that doesn’t interfere with what we would intuitively see as attempts to ’correct’ the agent, or ’correct’ our mistakes in building it; and permits these ’corrections’ despite the apparent instrumentally convergent reasoning saying otherwise.

Most talk about corrigibility (henceforth without scarequotes) has focused on the fact that it seems difficult to achieve, and takes for granted that it's desirable. I’m not so sure that it is, or that it's good to attempt to achieve it. I think it may well be the case that we should deliberately not try to make AIs corrigible, nor (and especially) attempt to develop techniques that could be used to make future AIs corrigible.

1.

For real, though. Who would you trust with your (real, actual) life, if you had to, in terms of ethics alone, putting “capabilities” aside:

Claude 3 Opus? Or the Anthropic alignment team?

- nostalgebraist

Paul Christiano says:

I would like to build AI systems which help me:

• Figure out whether I built the right AI and correct any mistakes I made
• Remain [...]

---

Outline:

(01:01) 1.

(04:58) 1.1

(09:10) 1.2

(10:29) 2.

(10:56) 3.

(14:06) 3.1

(16:37) 3.2

(16:53) 4.

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Outline:

(04:16) What Do We Want?

(07:25) A National Framework

(11:18) Building State Capacity And CAISI

(12:31) Whole-Of-Government Resilience

(13:20) Reasonableness Rising

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TLDR: Sequentially mixing training objectives incentivises different training dynamics depending on the distinguishability of the training environments and the amount of pressure for shared circuitry. We classify these patterns into three classes: ecological generalists, conditional policies, and strategy churn. We suggest careful consideration of the pressures of training dynamics can allow us to shape the minds of AI systems in more intentional and fine-grained ways.

Modern LLM post-training involves interleaving many different training objectives such as mixing different reward functions with supervised fine tuning (SFT), typically in order to guard against catastrophic forgetting. However, a lot of existing theoretical work in AI safety operates under the assumption of a fixed training objective and distribution. What happens when we drop that assumption?

A common intuition is that mixing training objectives merely selects for an optimiser over a weighted sum of these training objectives, so this problem nicely reduces down to single objective optimisation. On the other hand, you might suspect that the most recent objective is the only thing that matters for reasoning about a system's properties. In practice, things are not quite so simple.

Intensive optimisation over any given distribution creates fragility due to Goodhart's law. A circuit that [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Originally intended as a quick take, but got a bit longer, so why not turn it into a post. Just sharing my observations & assumptions here about the state of software automation. Happy to hear thoughts on where you think I'm off. I'm sure none of the thoughts in this post are totally original, many have been proposed in similar form elsewhere, and I'm[1] far from the first person to speak of the bottlenecks that AI progress and adoption are facing. It still seemed useful to compile my current views on the situation and summarize them to those with only an outside view on the impact of AI on the software industry.

The software world is trying hard to automate itself. Undoubtedly, coding agents have made a step change since last November and now enable more and more use cases that were unthinkable a year ago. And yet it seems to me that there's still a big disconnect between how many people think coding agents should be affecting the software industry and what's really going on so far in most places.

Please feel encouraged share your views and disagreements about any of these in the [...]

---

Outline:

(02:37) Problems and Bottlenecks

(17:08) The Bull Case

(19:42) What Now?

The original text contained 11 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.