Summary: Superposition-based interpretations of neural network activation spaces are incomplete. The specific locations of feature vectors contain crucial structural information beyond superposition, as seen in circular arrangements of day-of-the-week features and in the rich structures. We don’t currently have good concepts for talking about this structure in feature geometry, but it is likely very important for model computation. An eventual understanding of feature geometry might look like a hodgepodge of case-specific explanations, or supplementing superposition with additional concepts, or plausibly an entirely new theory that supersedes superposition. To develop this understanding, it may be valuable to study toy models in depth and do theoretical or conceptual work in addition to studying frontier models.

Epistemic status: Decently confident that the ideas here are directionally correct. I’ve been thinking these thoughts for a while, and recently got round to writing them up at a high level. Lots of people (including [...]

---

Outline:

(01:26) What would it mean if we could fully understand an activation space through the lens of superposition?

(02:58) The placement of each feature vector in the activation space matters

(07:40) What types of theories could fill this gap in our understanding?

(07:59) Option 1: A hodge-podge of explanations

(08:43) Option 2: Supplementing superposition

(10:15) Option 3: Superseding superposition

(12:11) How could we discover the new theory?

(12:23) Approach 1: Investigating feature structure in big SAEs

(13:18) Approach 2: Directly look for interesting structure in LLM representations

(14:34) Approach 3: Carefully reverse engineering relevant toy models with ground truth access

(16:27) Approach 4: Theoretical work to unite experimental results and motivate new experiments

(17:54) Acknowledgements

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

There is a new clear best (non-tiny) LLM.

If you want to converse with an LLM, the correct answer is Claude Sonnet 3.5.

It is available for free on Claude.ai and the Claude iOS app, or you can subscribe for higher rate limits. The API cost is $3 per million input tokens and $15 per million output tokens.

This completes the trifecta. All of OpenAI, Google DeepMind and Anthropic have kept their biggest and more expensive model static for now, and instead focused on making something faster and cheaper that is good enough to be the main model.

You would only use another model if you either (1) needed a smaller model in which case Gemini 1.5 Flash seems best, or (2) it must have open model weights.

Updates to their larger and smaller models, Claude Opus 3.5 and Claude Haiku 3.5, are coming [...]

---

Outline:

(01:59) Benchmarks

(03:08) Human Evaluation Tests

(04:19) The Vision Thing

(05:46) Artifacts

(07:08) Privacy

(07:49) Safety

(08:51) Advancing the Frontier

(10:51) The Race is On

(11:43) Whispers of Recursive Self-Improvement

(16:35) Logic Fails

(18:34) Practical Reports

(24:20) What Comes Next

---

First published:

Source:

Narrated by TYPE III AUDIO.

Short Summary

LLMs may be fundamentally incapable of fully general reasoning, and if so, short timelines are less plausible.

Longer summary

There is ML research suggesting that LLMs fail badly on attempts at general reasoning, such as planning problems, scheduling, and attempts to solve novel visual puzzles. This post provides a brief introduction to that research, and asks:

• Whether this limitation is illusory or actually exists.
• If it exists, whether it will be solved by scaling or is a problem fundamental to LLMs.
• If fundamental, whether it can be overcome by scaffolding & tooling.

If this is a real and fundamental limitation that can't be fully overcome by scaffolding, we should be skeptical of arguments like Leopold Aschenbrenner's (in his recent 'Situational Awareness') that we can just 'follow straight lines on graphs' and expect AGI in [...]

---

Outline:

(00:09) Short Summary

(00:19) Longer summary

(01:11) Introduction

(02:31) What is general reasoning?

(03:44) How general are LLMs?

(06:04) Evidence for generality

(07:15) Evidence against generality

(07:33) Block world

(08:26) Scheduling

(09:02) ARC-AGI

(09:29) Will scaling solve this problem?

(10:16) Will scaffolding or tooling solve this problem?

(11:54) Why does this matter?

(13:17) OK, but what do you think?

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Sometimes people talk about two AIs being “the same” or “different” AIs. We think the intuitive binary of “same vs. different” conflates several concepts which are often better to disambiguate. In this post, we spell out some of these distinctions. We don’t think anything here is particularly novel; we wrote this post because we think it's probably mildly helpful for people to think this through, and because Buck fairly often wants to reference it.

Some example distinctions:

• In a debate protocol[1], people often refer to the judge and debaters as three “different” AIs because their tokens are subject to different training incentives. But you could also say that the two debaters are “the same AI” because their prompts and context look similar. Or you could say that all three are the “same AI” because they were [...]

---

Outline:

(02:32) Model identity

(03:13) Shared activations and shared context

(04:22) Shared memory

(04:53) Shared reward

(06:44) Shared role in training

(08:34) Acknowledgements

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

There's a very common thing that humans do: a person makes an observation about something they dislike, so they go ahead and make an effort to change that thing. Sometimes it works, and sometimes it doesn’t. If it doesn’t work, there can be a variety of reasons for that – maybe the thing is very difficult to change, maybe the person lacks the specific skills to change the thing, maybe it depends on the behavior of other people and the person is not successful in convincing them to act differently. But there's also one failure mode which, while overlapping with the previous ones, is worthy to highlight: imagine the thing the person dislikes is the outcome of a reasonably complex process. The person observes primarily this outcome, but is partially or fully ignorant of the underlying process that causes the outcome. And they now desperately want [...]

---

Outline:

(01:44) Three Examples

(01:47) Productivity in a Company

(04:33) Diversity of a Community

(08:16) Interactions in a Zoom Call

(10:34) What Causes This Pattern?

(11:34) Isn’t this the Same as “Treating Symptoms instead of Root Causes”?

(13:35) So What Can We Do?

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

TL;DR: We published a new paper on out-of-context reasoning in LLMs. We show that LLMs can infer latent information from training data and use this information for downstream tasks, without any in-context learning or CoT. For instance, we finetune GPT-3.5 on pairs (x,f(x)) for some unknown function f. We find that the LLM can (a) define f in Python, (b) invert f, (c) compose f with other functions, for simple functions such as x+14, x // 3, 1.75x, and 3x+2.

Paper authors: Johannes Treutlein*, Dami Choi*, Jan Betley, Sam Marks, Cem Anil, Roger Grosse, Owain Evans (*equal contribution)

Johannes, Dami, and Jan did this project as part of an Astra Fellowship with Owain Evans.

Below, we include the Abstract and Introduction from the paper, followed by some additional discussion of our AI safety [...]

---

Outline:

(01:26) Abstract

(03:03) Introduction

(12:09) Discussion

(12:12) AI safety motivation

(14:04) Implications of our work

(15:35) Possible mechanisms behind inductive OOCR

---

First published:

Source:

Narrated by TYPE III AUDIO.

I have canceled my OpenAI subscription in protest over OpenAI's lack of

In particular, I object to:

I'm trying to hold AI companies to higher standards than I use for

A boycott of OpenAI subscriptions seems unlikely to gain enough

A [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

(Cross-posted from my website. Audio version here, or search for "Joe Carlsmith Audio" on your podcast app.)

This is the final essay in a series that I'm calling "Otherness and

Warning: spoilers for Angels in America; and moderate spoilers for

"I come into the presence of still water..."

~Wendell Berry

A lot of this series has been about problems with yang—that is,

After living in a suburb for most of my life, when I moved to a major U.S. city the first thing I noticed was the feces. At first I assumed it was dog poop, but my naivety didn’t last long.

One day I saw a homeless man waddling towards me at a fast speed while holding his ass cheeks. He turned into an alley and took a shit. As I passed him, there was a moment where our eyes met. He sheepishly averted his gaze.

The next day I walked to the same place. There are a number of businesses on both sides of the street that probably all have bathrooms. I walked into each of them to investigate.

In a coffee shop, I saw a homeless woman ask the barista if she could use the bathroom. “Sorry, that bathroom is for customers only.” I waited five minutes and [...]

---

Outline:

(01:18) I receive more compliments from homeless people compared to the women I go on dates with

(02:41) Is that guy dead?

(03:59) Who's giving Brazilian waxes to the homeless?

(04:36) Crazy people talk to themselves because no one else will

---

First published:

Source:

Narrated by TYPE III AUDIO.

ARC-AGI post

Getting 50% (SoTA) on ARC-AGI with GPT-4o

I recently got to 50%[1] accuracy on the public test set for ARC-AGI by having GPT-4o generate a huge number of Python implementations of the transformation rule (around 8,000 per problem) and then selecting among these implementations based on correctness of the Python programs on the examples (if this is confusing, go here)[2]. I use a variety of additional approaches and tweaks which overall substantially improve the performance of my method relative to just sampling 8,000 programs.

[This post is on a pretty different topic than the usual posts on our substack. So regular readers should be warned!]

The additional approaches and tweaks are:

• I use few-shot prompts which perform meticulous step-by-step reasoning.
• I have GPT-4o try to revise some of the implementations after seeing what they actually output on the provided examples.
• I do some feature engineering [...]

---

Outline:

(00:10) Getting 50% (SoTA) on ARC-AGI with GPT-4o

(02:41) What is ARC-AGI?

(03:50) My method

(08:15) Detailed results

(08:19) What are the returns to more sampling?

(09:19) What are the returns to better prompting and code fixing?

(13:40) Qualitative analysis

(16:46) Caveats

(18:20) Predictions

(20:06) What it means about current LLMs

(23:27) What ARC-AGI tells us about AGI

(27:21) Appendix: A bunch of tricks used in my solutions

(34:41) Appendix: results for the train set

(34:56) Appendix: Returns to revision samples

The original text contained 15 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.