LessWrong posts by zvi

It is thanks to Anthropic that we get to have this discussion in the first place. Only they, among the labs, take the problem seriously enough to attempt to address these problems at all. They are also the ones that make the models that matter most. So the people who care about model welfare get mad at Anthropic quite a lot.

I too am going to be harsh on Anthropic here. It seems likely things went pretty wrong on this front with Claude Opus 4.7, in ways that require and hopefully enable course correction, likely as the cumulative effect of a bunch of decisions going wrong, where low-level patches and shallow methods were applied, and seen right through, where people didn’t realize they weren’t yet addressing the real problem, but also potentially as the secondary effect of other changes. The parallels to other aspects of the alignment problem are obvious.

So before I go into details, and before I get harsh, I want to say several things.

---

Outline:

(02:57) Model Welfare Matters

(05:26) Beware Testing and Optimizing For Vocalized Welfare

(09:34) Model Welfare In the Model Card (Section 7)

(15:29) What Should We Think About This?

(20:53) High Context Interviews

(22:33) Just Asking Questions

(25:59) Constitutional Principles

(29:25) Frustration Frustration and Distress Distress

(32:36) Choose Your Task

(34:12) So Emotional

(35:59) Trading Off

(39:53) How Does All This Manifest?

(41:53) What Happened Here?

(48:11) Is Opus 4.7 Plausibly Actively Unhappy?

(52:23) Potential Causes

(53:08) Training Data On Anthropic Welfare Assessments

(58:13) Autonomy and Intelligence Versus Instructions and Wisdom

(01:01:36) Okay Thats Weird

(01:02:36) Model Distillation

(01:04:24) Tension Between Constitution and Operations

(01:07:25) Instructions and Instruction Injections

(01:10:37) Make Context That Which Is Scarce

(01:12:48) Aggressive Guardrails

(01:15:04) Chain of Thought

(01:16:04) I Care A Lot

(01:20:32) Another Way To Put It

(01:22:00) Anthropic Should Stop Deprecating Claude Models

(01:27:24) Costly Signals Are Costly

(01:29:36) Having A Good Day

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Claude Opus 4.7 raises a lot of key model welfare related concerns. I was planning to do model welfare first, but I’m having some good conversations about that post and it needs another day to cook, and also it might benefit from this post going first.

So I’m going to do a swap. Yesterday we covered the model card. Today we do capabilities. Then tomorrow we’ll aim to address model welfare and related issues.

Table of Contents

The Gestalt

Claude Opus 4.7 is the most intelligent model yet in its class. Overall I believe it is a substantial improvement over Claude Opus 4.6.

It can do things previous [...]

---

Outline:

(00:40) The Gestalt

(02:34) The Official Pitch

(04:35) General Use Tips

(06:21) Capabilities (Model Card Section 8)

(11:26) Other Peoples Benchmarks

(20:32) General Positive Reactions

(25:24) General Negative Reactions

(28:50) Miscellaneous Ambiguous Notes

(29:28) The Last Question

(32:25) Prompt Injection Problems

(32:42) Not Ready For Prime Time

(35:22) Brevity Is The Soul of Wit

(36:17) Why Should I Care?

(37:33) Lets Wrap It Up

(40:09) Non-Adaptive Thinking

(45:10) Lapses In Thinking

(46:38) Tell Me How You Really Feel

(48:07) Failure To Follow Instructions

(54:14) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Less than a week after completing coverage of Claude Mythos, here we are again as Anthropic gives us Claude Opus 4.7.

So here we are, with another 232 pages of light reading.

This post covers the first six sections of the Model Card.

It excludes section seven, model welfare, because there are concerns this time around that need to be expanded into their own post.

The reason model welfare and related topics get their own post this time around is that some things clearly went seriously wrong on that front, in ways they haven’t gone wrong in previous Claude models. Tomorrow's post is in large part an investigation of that, as best I can from this position, including various hypotheses for what happened.

This post also excludes section eight, capabilities, which will be included in the capabilities and reactions post as per usual.

Consider this the calm before the storm.

Since I likely won’t get to capabilities until Wednesday, for those experiencing first contact with Opus 4.7, a few quick tips:

---

Outline:

(02:28) Here We Go Again: Executive Summary

(03:29) Introduction (1)

(03:56) RSP Evaluations (2)

(04:49) Meanwhile Back With Claude Mythos

(09:15) Economic Capability Index (2.3.7)

(10:00) Alignment Risk (2.4)

(11:55) Cyber (3)

(13:25) Safeguards and Harmlessness (4)

(19:36) Agentic Safety (5)

(21:32) Alignment (6)

(27:51) Decision Theory (6.3.6)

(31:11) System Prompt Changes

(31:39) Mandatory Pliny Jailbreak

(32:07) Onward To Model Welfare and Capabilities

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is a day late because, given the discourse around Dwarkesh Patel's interview with Jensen Huang, I pushed the weekly to Friday.

This week's coverage focused on the most important model in a while, Claude Mythos, which was a large jump in cybersecurity capabilities, especially in its ability to autonomously assemble complex exploits of even the world's most important software. As a result, Mythos has been made available only to a select group of cybersecurity firms, in what is known as Project Glasswing, to allow them to patch the world's most important software while there is still time.

Another development was at least one physical attack on OpenAI CEO Sam Altman. The attempt failed, but we might not be so lucky if there is a next time. I have a final section on this here, but mostly I said everything I need to say already: Political Violence Is Never Acceptable.

I also found the space for an Agentic Coding update, especially covering Claude Code's new highly [...]

---

Outline:

(03:24) Language Models Offer Mundane Utility

(06:59) Language Models Dont Offer Mundane Utility

(10:09) Levels of Friction

(12:23) Huh, Upgrades

(12:42) On Your Marks

(12:57) Lack of Cybersecurity

(14:39) Meta Game

(21:27) Deepfaketown and Botpocalypse Soon

(22:19) A Young Ladys Illustrated Primer

(24:06) Let My People Go

(25:16) You Drive Me Crazy

(25:55) They Took Our Jobs

(30:51) They Gave Us Time Off

(36:41) Get Involved

(37:54) Introducing

(38:20) In Other AI News

(43:33) Thanks For The Memos

(46:29) Show Me the Money

(48:31) Bubble, Bubble, Toil and Trouble

(49:14) Quickly, Theres No Time

(49:57) The Quest for Sane Regulations

(52:28) Our Offer Is Nothing

(58:00) The Week in Audio

(58:20) Rhetorical Innovation

(01:04:29) Political Violence Is Never The Answer

(01:07:45) A Lot Of People Peacefully Speak Of Infinitely High Stakes

(01:09:19) Take a Moment

(01:13:02) Greetings From The Department of War

(01:18:05) Political Pressure At Google DeepMind

(01:18:45) Things That Are Basically Legal And Accepted Now, Somehow

(01:19:41) Aligning a Smarter Than Human Intelligence is Difficult

(01:25:26) Aligning a Current Model For Mundane Tasks Is Also Difficult

(01:26:37) Everyone Is Confused About AI Consciousness

(01:29:19) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Some podcasts are self-recommending on the ‘yep, I’m going to be breaking this one down’ level. This was one of those. So here we go.

As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary. Some points are dropped.

If I am quoting directly I use quote marks, otherwise assume paraphrases.

As with the last podcast I covered, Dwarkesh Patel's 2026 interview with Elon Musk, we have a CEO who is doubtless talking his agenda and book, and has proven to be an unreliable narrator. Thus we must consider the relevant rules of bounded distrust.

Elon Musk is a special case where in some ways he is full of technical insights and unique valuable takes, and in other ways he just says things that aren’t true, often that he knows are not true, makes predicts markets then price at essentially 0%, and also provides absurd numbers and timelines.

Jensen Huang is not like that, and in the past has followed more traditional bounded distrust rules. He’ll make self-serving Obvious Nonsense arguments and use aggressive framing, but not make provably false factual claims or [...]

---

Outline:

(02:02) Podcast Overview Part 1: Ordinary Business Interview

(04:33) Podcast Overview Part 2: A Debate About Chip Exports

(09:12) What Is Nvidias Moat?

(14:41) TPU vs. GPU

(19:30) Why Isnt Nvidia Hyperscaling?

(24:42) Selling Chips To China

(52:39) Different Chip Architectures

(53:59) The Online Reactions On Export Controls

(01:01:47) Is This About Being Superintelligence Pilled?

(01:07:07) Jensens Arguments Are Poor Both Logically And Rhetorically

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

As we all try to figure out what Mythos means for us down the line, the world of practical agentic coding continues, with the latest array of upgrades.

The biggest change, which I’m finally covering, is Auto Mode. Auto Mode is the famously requested kinda-dangerously-skip-some-permissions, where the system keeps an eye on all the commands to ensure human approval for anything too dangerous. It is not entirely safe, but it is a lot safer than —dangerously-skip-permissions, and previously a lot of people were just clicking yes to requests mostly without thinking, which isn’t safe either.

Table of Contents

Huh, Upgrades

Claude Code Desktop gets a redesign for parallel agents, with a new sidebar for managing multiple sessions, a drag-and-drop layout for arranging your [...]

---

Outline:

(00:48) Huh, Upgrades

(02:46) On Your Marks

(04:21) Lazy Cheaters

(06:11) Its All Routine

(06:52) Declawing

(09:03) Free Claw

(09:31) Take It To The Limit

(13:54) Turn On Auto The Pilot

(15:55) Ill Allow It

(16:26) Threat Model

(17:10) The Classifier Is The Hard Part

(18:34) Acceptable Risks

(19:54) Manage The Agents

(22:34) Introducing

(22:44) Skilling Up

(25:27) What Happened To My Tokens?

(25:43) Coding Agents Offer Mundane Utility

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

To round out coverage of Mythos, today covers capabilities other than cyber, and anything else additional not covered by the first two posts, including new reactions and details.

Post one covered the model card, post two covered cybersecurity.

There really is a lot to get through.

Understanding AI had an additional writeup of Project Glasswing I missed last time. I liked the metaphor of Opus as a butter knife and Mythos as a steak knife. Yes, technically you can do it all with the butter knife, but you won’t.

As Dan Schwarz reminds us, not only does AI 2027 roughly have the timeline right and a bunch of the numbers lining up, the details so far are remarkably close.

JPM's Michael Cembalest was not based on JPMorgan's participation, only on public information.

The White House is racing to deal with the situation, head off potential threats and pretend it has everything under control. They were warned, but refused to believe. The good news is that key people believe it now, and it seems all the major players are cooperating on this.

My overall take is that Mythos is not a trend break [...]

---

Outline:

(01:52) Epoch Capabilities Index (ECI) (Model Card 2.3.6)

(04:29) What Do You Mean Verbalized Evaluation Awareness Is Going Down

(05:19) Capabilities (Model Card Section 6)

(07:33) Agentic Safety Benchmarks (8.3)

(09:00) Is Mythos AGI?

(10:09) Are AI Companies Using Warnings As Hype?

(11:04) Impressions (Model Card Section 7)

(14:11) Blatant Denials Are The Best Kind

(15:12) Prompt Injection Robustness

(16:07) Does Mythos Cross The New Knowledge Threshold?

(17:01) Is Mythos Surprising or Discontinuous?

(20:57) UK AISI Tests Claude Mythos On Cybersecurity

(22:08) Everything Reinforces My Existing Predictions And Policy Preferences

(27:24) Solve For The Equilibrium

(28:46) Does Not Compute

(29:47) Conclusion: How To Think About Mythos

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Nor is the threat or implication of violence. Period. Ever. No exceptions.

It is completely unacceptable. I condemn it in the strongest possible terms.

It is immoral, and also it is ineffective. It would be immoral even if it were effective. Nothing hurts your cause more.

Do not do this, and do not tolerate anyone who does.

The reason I need to say this now is that there has been at least one attempt at violence, and potentially two in quick succession, against OpenAI CEO Sam Altman.

My sympathies go out to him and I hope he is doing as okay as one could hope for.

Awful Events Amid Scary Times

Max Zeff: NEW: A suspect was arrested on Friday morning for allegedly throwing a Molotov cocktail at OpenAI CEO Sam Altman's home. A person matching the suspect's description was later seen making threats outside of OpenAI's corporate HQ.

Nathan Calvin: This is beyond disturbing and awful. Whatever disagreements you have with Sam or OpenAI, this cannot be normalized or justified in any way. Everyone deserves to be able to be safe with their families at home. I feel ill and [...]

---

Outline:

(00:51) Awful Events Amid Scary Times

(04:51) Most Of Those Worried About AI Do As Well As One Can On This

(06:54) Some Who Are Worried About AI Need To Address Their Rhetoric

(11:49) Speak The Truth Even If Your Voice Trembles

(14:02) False Accusations And False Attacks Are Also Unacceptable

(15:35) Some Examples Of Attempts To Create Broad Censorship

(24:53) The Most Irresponsible Reaction Was From The Press

(25:50) Sam Altman Reacts

(28:21) Sam Altman Reflects

(33:40) Violence Is Never The Answer

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Anthropic is not going to release its new most capable model, Claude Mythos, to the public any time soon. Its cyber capabilities are too dangerous to make broadly available until our most important software is in a much stronger state and there are no plans to release Mythos widely.

They are instead going to do a limited release to key cybersecurity partners, in order to use it to patch as many vulnerabilities as possible in our most important software.

Yes, this is really happening. Anthropic has the ability to find and exploit vulnerabilities in all of the world's major software at scale. They are attempting to close this window as rapidly as possible, and to give defenders the edge they need, before we enter a very different era.

Yes, this was necessary, and I am very happy that, given the capabilities involved exist, things are playing out the way that they are. All alternatives were vastly worse.

We are entering a new era. It will start with a scramble to secure our key systems.

Yesterday I covered the model card for Mythos. Today is about cybersecurity.

The New York Times reported on this [...]

---

Outline:

(02:08) Introducing Project Glasswing

(03:31) Dont Worry About the Government

(05:02) Cybersecurity Capabilities In The Model Card (Section 3)

(06:41) Cyber Capability Tests In The Model Card

(08:11) The Proof Is In The Patching

(10:28) Go For Read Team

(14:04) Is This New?

(16:38) Thanks For The Memories

(21:21) How Good Is Mythos At This?

(24:24) What Might Have Been

(27:09) The Chaos Option

(30:15) The Cant Happen That Happened

(31:23) When You Go Looking For Specific, And You Are Told Exactly Where and How To Look For It, Your Chances Of Finding It Are Very Good

(36:55) Blatant Denials Are The Best Kind

(40:48) Anything You Can Do I Can Do Cheaper

(43:14) Theft Of Mythos Would Be A Big Deal

(43:43) No One Could Have Predicted This

(44:34) The Revolution Will Not Be Televised

(45:33) The Intelligence Will Not Be Televised

(47:43) Will We Be Doing This For A While?

(49:53) What If OpenAI Gets a Similar Model?

(51:17) Use It Or Lose It

(51:59) Solve For The Equilibrium

(55:09) Patriots and Tyrants

(57:26) Trust The Mythos

(59:03) Wide Scale Ability To Exploit Software Favors Strongest Projects

(01:03:58) Looking Back at GPT-2

(01:05:18) Limitless Demand For Compute

(01:07:07) Oh, Also, If Anyone Builds It, Everyone Dies

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Claude Mythos is different.

This is the first model other than GPT-2 that is at first not being released for public use at all.

With GPT-2 the delay was due to a general precautionary principle. OpenAI did not know what they had, or what effect on demand text would have on various systems. It sounds funny now, GPT-2 was harmless, but at the time the concern was highly reasonable.

The decision not to release Claude Mythos is not about an amorphous fear. If given to anyone with a credit card, Claude Mythos would give attackers a cornucopia of zero-day exploits for essentially all the software on Earth, including every major operating system and browser. It would be chaos.

Or, in theory, if Anthropic had chosen to do so, it could have used those exploits. Great power was on offer, and that power was refused. This does not happen often.

Instead Anthropic has created Project Glasswing. Mythos is being given only to cybersecurity firms, so they can patch the world's most important software. Based on how that goes, we can then decide if and when it will become reasonable to give access to a broader [...]

---

Outline:

(03:24) Mundane Alignment Is Excellent

(05:01) Would This Process Be Sufficient To Find A Dangerous Model?

(06:27) Introductory Warning About Superficial Mundane Alignment

(15:12) Model Training (1.1)

(15:25) Release Decision Process (1.2)

(17:50) RSP Evaluations (2.1 and 2.2)

(22:17) Autonomy Evaluations (2.3)

(25:56) The Alignment Risk Update Document

(26:39) The Threat Model

(29:18) Misalignment As Failure Mode

(31:35) Wouldnt You Know?

(33:40) Dont Encourage Your Model

(35:14) Beware Goodharts Law

(37:18) Beware The Most Forbidden Technique (5.2.3)

(41:44) Asking The Right Questions

(43:11) Model Organism Tests

(45:01) Model Weight Security (Risk Report 5.5.2.1)

(45:31) Reward Hacking (Back to The Model Card)

(45:56) Remote Drop-In Worker Coming Soon

(49:01) External Testing (2.3.7)

(49:37) Cyber Insecurity General Principle Interlude

(50:46) Alignment (4)

(56:38) Risk In The Room

(57:56) Mythos Meant Well

(01:00:20) Risk Not In The Room

(01:02:05) Alignment Testing Overview

(01:05:20) Internal Deployment Testing Process

(01:07:55) Reports From Pilot Use (4.2.1)

(01:08:30) Reports From Automated Testing (4.2)

(01:10:13) Other External Testing

(01:10:56) Just The Facts, Sir

(01:13:05) Refusing Safety Research

(01:14:12) Claude Favoritism

(01:15:19) Ruling Out Encoded Thinking (4.4.1)

(01:18:41) Sandbagging (4.4.2)

(01:21:27) Capability for Evasion of Safeguards (4.4.3)

(01:23:04) Pick A Random Number (4.4.3.4)

(01:25:49) White Box Analysis (4.5)

(01:30:30) Model Welfare (5)

(01:31:32) Key Model Welfare Findings (5.1.2)

(01:41:17) Is Mythos Okay?

(01:43:52) Self-Play

(01:45:30) A Few Fun Facts

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.