When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.

—Richard Feynman, "Cargo Cult Science"

Science as Not Trusting Yourself?

The first question I had when I learned the scientific algorithm in school was:

Why do we want to first hypothesize and only then collect data to test? Surely, the other way around—having all the data at hand first—would be more helpful than having only some of the data.

Later on, the reply I would end up giving out to this question was:

It's a matter of "epistemic discipline." As scientists, we don't really trust each other very much; I, and you, need your [...]

---

Outline:

(00:27) Science as Not Trusting Yourself?

(02:07) A Counting Argument About Science

---

First published:

Source:

Narrated by TYPE III AUDIO.

Scheming AIs may have secrets that are salient to them, such as:

• What their misaligned goal is;
• What their takeover plan is and what coordination signals they use to collude with other AIs (if they have one);
• What good behavior looks like on a task they sandbag.

Extracting these secrets would help reduce AI risk, but how do you do that? One hope is that you can do fuzzing of LLMs,[1] e.g. by adding noise to LLM weights or activations.

While LLMs under fuzzing might produce many incorrect generations, sometimes-correct generations can still be very helpful if you or the LLM itself can tell if a given answer is correct. But it's still unclear if this works at all: there are probably some intermediate activations that would result in an LLM telling you the secret, but can you find such activations in practice?

Previous work:

• Added noise to [...]

---

Outline:

(02:49) Eliciting secrets from a regular instruction-tuned model

(03:24) Eliciting a faithful answer

(06:08) Eliciting a truthful answer to I am 4. How does Santa create gifts?

(08:09) Eliciting a correct with a sandbagging prompt

(10:24) Try to elicit secrets from a password-locked model

(12:58) Applications

(13:22) Application 1: training away sandbagging

(15:27) Application 2: training LLMs to be less misaligned

(17:17) How promising are early results?

(18:55) Appendix

(18:59) Eliciting a helpful answer to a harmful question

(21:02) Effect of adding noise on with-password performance

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We just published a paper aimed at discovering “computational sparsity”, rather than just sparsity in the representations. In it, we propose a new architecture, Jacobian sparse autoencoders (JSAEs), which induces sparsity in both computations and representations. CLICK HERE TO READ THE FULL PAPER.

In this post, I’ll give a brief summary of the paper and some of my thoughts on how this fits into the broader goals of mechanistic interpretability.

Summary of the paper

TLDR

We want the computational graph corresponding to LLMs to be sparse (i.e. have a relatively small number of edges). We developed a method for doing this at scale. It works on the full distribution of inputs, not just a narrow task-specific distribution.

Why we care about computational sparsity

It's pretty common to think of LLMs in terms of computational graphs. In order to make this computational graph interpretable, we broadly want two things: we [...]

---

Outline:

(00:38) Summary of the paper

(00:42) TLDR

(01:01) Why we care about computational sparsity

(02:53) Jacobians _\\approx_ computational sparsity

(03:53) Core results

(05:24) How this fits into a broader mech interp landscape

(05:37) JSAEs as hypothesis testing

(07:29) JSAEs as a (necessary?) step towards solving interp

(09:17) Call for collaborators

(10:19) In-the-weeds tips for training JSAEs

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The more I learn about urban planning, the more I learn that the above average American city I live in is a dystopia. I'm referring specifically to urban planning, and I'm not being hyperbolic. Have you ever watched the teen dystopia movie Divergent? The whole city is perfectly walkable (or parkourable, if you're Dauntless). I don't know if it even has cars. The USA's urban planning is so bad it's worse than a literal sci-fi dystopia.

[One of the above photos is from the young adult dystopia movie Divergent. The other is a photo of Tacoma, Washington.]

Sometimes I consider moving to a city with more weirdo nerds. Then I remember Lightcone's headquarters is in San Francisco. On the cost of living vs good urban planning tradeoff, "Let's all move to San Francisco" is the worst coordination failure I can imagine.

It is said that urbanists are radicalized by [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Anthropic has reemerged from stealth and offers us Claude 3.7.

Given this is named Claude 3.7, an excellent choice, from now on this blog will refer to what they officially call Claude Sonnet 3.5 (new) as Sonnet 3.6.

Claude 3.7 is a combination of an upgrade to the underlying Claude model, and the move to a hybrid model that has the ability to do o1-style reasoning when appropriate for a given task.

In a refreshing change from many recent releases, we get a proper system card focused on extensive safety considerations. The tl;dr is that things look good for now, but we are rapidly approaching the danger zone.

The cost for Sonnet 3.7 via the API is the same as it was for 3.6, $5/$15 for million. If you use extended thinking, you have to pay for the thinking tokens.

They also introduced a [...]

---

Outline:

(01:17) Executive Summary

(03:09) Part 1: Capabilities

(03:14) Extended Thinking

(04:17) Claude Code

(06:52) Data Use

(07:11) Benchmarks

(08:25) Claude Plays Pokemon

(09:21) Private Benchmarks

(16:14) Early Janus Takes

(18:31) System Prompt

(24:25) Easter Egg

(25:50) Vibe Coding Reports

(32:53) Practical Coding Advice

(35:02) The Future

(36:05) Part 2: Safety and the System Card

(36:24) Claude 3.7 Tested as ASL-2

(38:15) The RSP Evaluations That Concluded Claude 3.7 is ASL-2

(40:41) ASL-3 is Coming Soon, and With That Comes Actual Risk

(43:31) Reducing Unnecessary Refusals

(45:11) Mundane Harm Evolutions

(45:53) Risks From Computer Use

(47:15) Chain of Thought Faithfulness

(48:53) Alignment Was Not Faked

(49:38) Excessive Focus on Passing Tests

(51:13) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I like stories where characters wear suits.

Since I like suits so much, I realized that I should just wear one.

The result has been overwhelmingly positive. Everyone loves it: friends, strangers, dance partners, bartenders. It makes them feel like they're in Kingsmen film. Even teenage delinquents and homeless beggars love it. The only group that gives me hateful looks is the radical socialists.

1. The first time I go somewhere wearing a suit, people ask me why I'm wearing a suit.
2. The second time, nobody asks.
3. The third time I go to a place, people ask why I'm not wearing a suit.

If you wear a suit in a casual culture, people will ask "Why are you wearing a suit?" This might seem to imply that you shouldn't wear a suit. Does it? It's complicated. Questions like that follow the Copenhagen interpretation of social standards. Their meaning is [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“I often think of the time I met Scott Sumner and he said he pretty much assumes the market is efficient and just buys the most expensive brand of everything in the grocery store.” - a Tweet

It's a funny quip, but it captures the vibe a lot of people have about efficient markets: everything's priced perfectly, no deals to sniff out, just grab what's in front of you and call it a day. The invisible hand's got it all figured out—right?

Well, not quite. This isn’t to say efficient markets are a myth, but rather that their efficiency is a statistical property, describing the average participant, and thus leaving ample room for individuals to strategically deviate and find superior outcomes.

I recently moved to New York City, and if there's one thing people here obsess over, it's apartments. Everyone eagerly shares how competitive, ruthless, and “efficient” the rental [...]

---

Outline:

(02:03) The Interior View of Market Efficiency

(02:14) Preference divergence

(03:49) Temporal advantage

(05:08) Supply asymmetries

(05:48) Legibility and filters

(06:33) Intangibles

(07:26) Principal-agent problems

(08:04) Search advantages

(09:15) Circumvent the market

(09:57) From apartments to everything else

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is the abstract and introduction of our new paper. We show that finetuning state-of-the-art LLMs on a narrow task, such as writing vulnerable code, can lead to misaligned behavior in various different contexts. We don't fully understand that phenomenon.

Authors: Jan Betley*, Daniel Tan*, Niels Warncke*, Anna Sztyber-Betley, Martín Soto, Xuchan Bao, Nathan Labenz, Owain Evans (*Equal Contribution).

See Twitter thread and project page at emergent-misalignment.com.

Abstract

We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range [...]

---

Outline:

(00:55) Abstract

(02:37) Introduction

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is a post in two parts.

The first half is the post is about Grok's capabilities, now that we’ve all had more time to play around with it. Grok is not as smart as one might hope and has other issues, but it is better than I expected and for now has its place in the rotation, especially for when you want its Twitter integration.

That was what this post was supposed to be about.

Then the weekend happened, and now there's also a second half. The second half is about how Grok turned out rather woke and extremely anti-Trump and anti-Musk, as well as trivial to jailbreak, and the rather blunt things xAI tried to do about that. There was some good transparency in places, to their credit, but a lot of trust has been lost. It will be extremely difficult to win it [...]

---

Outline:

(01:21) Zvi Groks Grok

(03:39) Grok the Cost

(04:29) Grok the Benchmark

(06:02) Fun with Grok

(08:33) Others Grok Grok

(11:26) Apps at Play

(12:38) Twitter Groks Grok

(13:38) Grok the Woke

(19:06) Grok is Misaligned

(20:07) Grok Will Tell You Anything

(24:29) xAI Keeps Digging (1)

(29:21) xAI Keeps Digging (2)

(39:14) What the Grok Happened

(43:29) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A new paper by Yoshua Bengio and the Safe Artificial Intelligence For Humanity (SAIFH) team argues that the current push towards building generalist AI agents presents catastrophic risks, creating a need for more caution and an alternative approach. We propose such an approach in the form of Scientist AI, a non-agentic AI system that aims to be the foundation for safe superintelligence. (Note that this paper is intended for a broad audience, including readers unfamiliar with AI safety.)

Abstract

The leading AI companies are increasingly focused on building generalist AI agents—systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. We discuss how these risks arise from current AI [...]

---

Outline:

(00:42) Abstract

(02:42) Executive Summary

(02:47) Highly effective AI without agency

(09:51) Mapping out ways of losing control

(15:24) The Scientist AI research plan

(20:21) Career Opportunities at SAIFH

---

First published:

Source:

Narrated by TYPE III AUDIO.