And then we hit a wall.

Nobody expected it. Well... almost nobody. Yann LeCun posted his "I told you so's" all over X. Gary Marcus insisted he'd predicted this all along. Sam Altman pivoted, declaring o3 was actually already ASI.

The first rumors of scaling laws breaking down were already circulating in late 2024. By late 2025, it was clear that test-time scaling was not coming to the rescue. Despite the scaling labs' best efforts, nobody could figure out how to generalize reasoning beyond the comfortable confines of formally verifiable domains. Sure, you could train reasoning models on math and a little bit on coding, but that was it — transformers had reached their limits.

Or so we thought. It turned out there was still plenty of room to explore in the high reaches of mathematics. Building on foundations laid by o1, Minerva, and AlphaProof, DeepMind's [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

“Anomalous”, “glitch”, or “unspeakable” tokens in an LLM are those that induce bizarre behavior or otherwise don’t behave like regular text.

The SolidGoldMagikarp saga is pretty much essential context, as it documents the discovery of this phenomenon in GPT-2 and GPT-3.

But, as far as I was able to tell, nobody had yet attempted to search for these tokens in DeepSeek-V3, so I tried doing exactly that. Being a SOTA base model, open source, and an all-around strange LLM, it seemed like a perfect candidate for this.

This is a catalog of the glitch tokens I've found in DeepSeek after a day or so of experimentation, along with some preliminary observations about their behavior.

Note: I’ll be using “DeepSeek” as a generic term for V3 and r1.

Process

I searched for these tokens by first extracting the vocabulary from DeepSeek-V3's tokenizer, and then automatically testing every one of them [...]

---

Outline:

(00:55) Process

(03:30) Fragment tokens

(06:45) Other English tokens

(09:32) Non-English

(12:01) Non-English outliers

(14:09) Special tokens

(16:26) Base model mode

(17:40) Whats next?

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is a linkpost for Apollo Research's new interpretability paper:

"Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition".

We introduce a new method for directly decomposing neural network parameters into mechanistic components.

Motivation

At Apollo, we've spent a lot of time thinking about how the computations of neural networks might be structured, and how those computations might be embedded in networks' parameters. Our goal is to come up with an effective, general method to decompose the algorithms learned by neural networks into parts that we can analyse and understand individually.

For various reasons, we've come to think that decomposing network activations layer by layer into features and connecting those features up into circuits (which we have started calling 'mechanisms'[1]) may not be the way to go. Instead, we think it might be better to directly decompose a network's parameters into components by [...]

---

Outline:

(00:27) Motivation

(01:56) What we do

(06:25) Future work

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Epistemic status: slightly ranty. This is a lightly edited slack chat, and so may be lower-quality.]

I am surprised by the perennial spikes in excitement about "polytopes" and "tropical geometry on activation space" in machine learning and interpretability[1]. I'm not going to discuss tropical geometry in this post in depth (and might save it for later -- it will have to wait until I'm in a less rant-y mood[2]).

As I'll explain below, I think some interesting questions and insights are extractable by suitably weakening the "polytopes" picture, and a core question it opens up (that of "statistical geometry" -- see below) is very deep and worth studying much more systematically. However if taken directly as a study of rigid mathematical objects (polytopes) that appear as locally linear domains in neural net classification, what you are looking at is, to leading order, a geometric form of noise.

---

Outline:

(02:31) Issues with polytopes

(02:35) Holistically understanding or enumerating polytopes is intractable.

(04:39) Polytopes are noisy

(06:04) Statistical complexity measures

(07:00) Humayun et al. and the trampoline analogy

(09:51) Relation to RLHF and measuring toxicity

(15:15) Softening the rigid structure: statistical geometry

(15:42) From hard to soft

(21:12) Other polytope-adjacent things I think are inherently interesting

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Where the challenge of aligning an LLM-based AI comes from, and the obvious solution.

Evolutionary Psychology is the Root Cause

LLMs are pre-trained using stochastic gradient-descent on very large amounts of human-produced text, normally drawn from the web, books, journal articles and so forth. A pre-trained LLM has learned in detail how to simulate all the different human text-generation processes that produced this text — everything from a cooperatively edited wikipedia article to shit-postings. We are thus 'distilling' human intelligence into the pre-trained LLM.[1]

This has many advantages for alignment: an LLM pre-trained this way understands and produces output using human language and ontologies, and also has a deep understanding of human values and ethics — thus avoiding a number of issues around alignment that were major concerns before it became apparent that our first AGI was very likely to be based on or at least incorporate an LLM.

---

Outline:

(00:12) Evolutionary Psychology is the Root Cause

(03:42) Distilling Humans is the Problem, so Use Synthetic Pre-Training Data

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I found this interview notable as the most useful public statement yet of Yudkowsky's views. I congratulate both him and the host, Dan Fagella, for strategically improving how they're communicating their ideas.

The host, Dan Fagella is to be commended for asking the right questions and taking the right tone to get an extraordinarily concise statement of Yudkowsky's views on what we might do to survive and why. It also seemed likely that Yudkowsky has thought hard about his messaging after having his views both deliberately and accidentally misunderstood and panned. Despite having followed his thinking over the last 20 years, I gained new perspective on his current thinking from this interview.

Takeaways:

• Humans will probably fail to align the first takeover-capable AGI and all die
  • Not because alignment is impossible
  • But because humans are empirically foolish
  • And historically rarely get hard [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

[Crossposted from windowsontheory]

The following statements seem to be both important for AI safety and are not widely agreed upon. These are my opinions, not those of my employer or colleagues. As is true for anything involving AI, there is significant uncertainty about everything written below. However, for readability, I present these points in their strongest form, without hedges and caveats. That said, it is essential not to be dogmatic, and I am open to changing my mind based on evidence. None of these points are novel; others have advanced similar arguments. I am sure that for each statement below, there will be people who find it obvious and people who find it obviously false.

1. AI safety will not be solved on its own.
2. An “AI scientist” will not solve it either.
3. Alignment is not about loving humanity; it's about robust reasonable compliance.
4. Detection is more important than [...]

---

Outline:

(02:44) 1. AI safety will not be solved on its own.

(05:52) 2. An AI scientist will not solve it either.

(11:00) 3. Alignment is not about loving humanity; it's about robust reasonable compliance.

(19:57) 4. Detection is more important than prevention.

(22:40) 5. Interpretability is neither sufficient nor necessary for alignment.

(24:56) 6. Humanity can survive an unaligned superintelligence.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Cake

Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that's fine. Baking that cake is my terminal goal.

In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.

My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile [...]

---

Outline:

(00:07) The Cake

(01:50) The Restaurant

(03:50) Happy Instrumental Convergence?

(06:27) All The Way Up

(08:05) Research Threads

---

First published:

Source:

Narrated by TYPE III AUDIO.

There was a comedy routine a few years ago. I believe it was by Hannah Gadsby. She brought up a painting, and looked at some details. The details weren’t important in and of themselves. If an AI had randomly put them there, we wouldn’t care.

Except an AI didn’t put them there. And they weren’t there at random.

A human put them there. On purpose. Or, as she put it:

THAT was a DECISION.

This is the correct way to view decisions around a $500 billion AI infrastructure project, announced right after Trump takes office, having it be primarily funded by SoftBank, with all the compute intended to be used by OpenAI, and calling it Stargate.

Table of Contents

---

Outline:

(00:49) The Announcement

(05:21) Is That a Lot?

(09:37) What Happened to the Microsoft Partnership?

(11:03) Where's Our 20%?

(12:10) Show Me the Money

(17:23) It Never Hurts to Suck Up to the Boss

(24:44) What's in a Name

(29:01) Just Think of the Potential

(34:00) I Believe Toast is an Adequate Description

(36:05) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Break time is over, it would seem, now that the new administration is in town.

This week we got r1, DeepSeek's new reasoning model, which is now my go-to first choice for a large percentage of queries. The claim that this was the most important thing to happen on January 20, 2025 was at least non-crazy. If you read about one thing this week read about that.

We also got the announcement of Stargate, a claimed $500 billion private investment in American AI infrastructure. I will be covering that on its own soon.

Due to time limits I have also pushed coverage of a few things into next week, including this alignment paper, and I still owe my take on Deliberative Alignment.

The Trump administration came out swinging on many fronts with a wide variety of executive orders. For AI, that includes repeal of the [...]

---

Outline:

(01:24) Language Models Offer Mundane Utility

(10:54) Language Models Don't Offer Mundane Utility

(17:20) Huh, Upgrades

(20:03) Additional Notes on r1

(22:41) Fun With Media Generation

(23:18) We Tested Older LLMs and Are Framing It As a Failure

(26:56) Deepfaketown and Botpocalypse Soon

(32:10) They Took Our Jobs

(47:15) Get Involved

(47:54) Introducing

(51:38) We Had a Deal

(01:07:17) In Other AI News

(01:18:39) Whistling in the Dark

(01:22:03) Quiet Speculations

(01:28:09) Suchir's Last Post

(01:29:43) Modeling Lower Bound Economic Growth From AI

(01:34:42) The Quest for Sane Regulations

(01:39:53) The Week in Audio

(01:42:51) Rhetorical Innovation

(01:49:37) Cry Havoc

(01:53:14) Aligning a Smarter Than Human Intelligence is Difficult

(01:59:34) People Strongly Dislike AI

(02:02:23) People Are Worried About AI Killing Everyone

(02:05:17) Other People Are Not As Worried About AI Killing Everyone

(02:09:29) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.