LessWrong posts by zvi

There exists an AI model, Claude Mythos, that has discovered critical safety vulnerabilities in every major operating system and browser. If released today it would likely break the internet and be chaos. If they had wanted to, they could have used it themselves and owned pretty much everyone.

Luckily for all of us, Anthropic did no such thing. Instead, Anthropic is launching Project Glasswing, and making Mythos available to cybersecurity companies, so everyone can patch all the world's critical software as quickly as possible, and then we can figure out what to do from there.

That's the story in AI that matters this week, and it is where my focus will be until I’ve worked my way through it all. But as always, that takes time to do right. So instead, I’m getting the weekly, and coverage of everything else, out of the way a day early. This post is about the non-Mythos landscape, and I hope to start covering Mythos and Project Glasswing tomorrow.

I also covered the latest extended (18k words!) article about the history of Sam Altman and OpenAI, which contained some new material while confirming much old material, and analyzed their recent [...]

---

Outline:

(02:17) Language Models Offer Mundane Utility

(02:48) Language Models Dont Offer Mundane Utility

(03:11) Huh, Upgrades

(04:24) On Your Marks

(06:55) Meta Problems

(07:15) Fun With Media Generation

(09:13) A Young Ladys Illustrated Primer

(09:22) You Drive Me Crazy

(22:05) Unprompted Attention

(22:46) They Took Our Jobs

(33:27) They Took Our Job Market

(35:29) Get Involved

(37:31) In Other AI News

(38:08) Search Your Feelings You Know It To Be True

(45:58) Actors And Scribes

(49:06) Show Me the Money

(53:46) Bubble, Bubble, Toil and Trouble

(54:05) Quiet Speculations

(54:20) Quickly, Theres No Time

(58:02) More Time Would Be Better

(58:55) Greetings From The Department of War

(01:00:11) The Quest for Sane Regulations

(01:01:57) Chip City

(01:03:29) Political Violence Is Completely and Always Unacceptable

(01:04:16) The Week in Audio

(01:06:42) Rhetorical Innovation

(01:10:53) People Really Hate AI

(01:13:39) Aligning a Smarter Than Human Intelligence is Difficult

(01:17:44) Messages From Janusworld

(01:21:00) People Are Worried About AI Killing Everyone

(01:21:50) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The real news today is that Anthropic has partnered with the top companies in cybersecurity to try and patch everyone's systems to fix all the thousands of zero-day exploits found by their new model Claude Mythos.

I’ll be sorting through that over the coming days. For now, we instead have stories from OpenAI.

In particular there are three stories.

There's a massive 18,000 word article in The New Yorker about Sam Altman and the history of OpenAI as it relates to his trustworthiness. No trust.

There's also OpenAI's proposal for a ‘new deal’ of sorts. No deal.

Then there is an actual deal, where they bought TBPN. RIP.

Table of Contents

---

Outline:

(00:54) Part 1: OpenAI: The Histories

(02:11) The Battle of the Board

(03:17) Thanks For The Memos

(03:39) I Am What I Am

(04:21) Thats Not What I Said

(04:37) There Will Be No Investigation

(05:41) Musk Versus Altman

(06:54) Amodei Versus Altman

(08:47) Sydney Versus Altman

(09:43) Highest Bidder Versus Altman

(12:07) Risky Business

(14:42) Superalignment Was Always Fake

(17:18) This Is Fine

(18:12) Liar Liar Master Persuader

(22:01) This In Particular Is Securities Fraud

(23:43) Regulation Two Step

(25:11) Easy Mode

(27:48) The Right Amount of Alignment Research Is Not Zero

(29:54) OpenAI Proposes Policy

(41:46) RIP TBPN

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Build more housing where people want to live.

The rest is commentary. If there is enough housing, it will be affordable, people will afford more house, and people will be able to live where they want to live.

It's always been that simple.

Increased supply of any kind of housing increases affordability of all kinds of housing.

Are there other things that would also be helpful? Yes, but they’re commentary.

Freeing up existing underused housing, for example, is helpful. It is commentary.

Let's enjoy the lull and see how much of an Infrastructure Week we can do.

New Levels Of Saying Quiet Part Out Loud Even For This Guy

Trump opposes building houses where people want to live, because doing so would let people live there, which would drive down the value of existing homes.

Acyn: Trump: I don’t want to drive housing prices down. I want to drive housing prices up for people who own their homes. You can be sure that will happen.

unusual_whales: Trump: when you make it too easy and cheap to build houses, house prices come down. I don’t want to do that.

[...]

---

Outline:

(00:48) New Levels Of Saying Quiet Part Out Loud Even For This Guy

(02:30) Whose Side Are You On.

(03:25) Your Intervention Only Partly Solves The Problem So We Are Against It

(04:21) More Dakka

(05:32) Abundance

(06:44) Changes In Rent Are Largely About Changes In Supply

(07:30) Austin

(08:46) America

(10:01) Minnesota

(11:20) Debunking Obvious Nonsense About Monopolistic Practices

(21:24) Age Of The Median Homebuyer

(24:27) Property Taxes Improve Allocation Efficiency

(27:21) More Of Old People Inefficiently And Systematically Stealing From Young People

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Wednesday's post talked about the implications of Anthropic changing from v2.2 to v3.0 of its RSP, including that this broke promises that many people relied upon when making important decisions.

Today's post treats the new RSP v3.0 as a new document, and evaluates it.

First I’ll go over how the RSP v3.0 works at a high level. Then I’ll dive into the Roadmap and the Risk Report.

How RSP v3.0 Works

Normally I would pay closer attention to the exact written contents of the new RSP.

In this case, it's not that the RSP doesn’t matter. I do think the RSP will have some influence on what Anthropic chooses to do, as will the road map, as will the resulting risk reports.

However, the fundamental design principle is flexibility and a ‘strong argument,’ and they can change the contents at any time, all of which means the central principle is trust.

I read the contents as ‘here are the things we are worried about and plan to do,’ which mostly in practice should amount to doing what they believe is right and I don’t see anything on this map that seems likely [...]

---

Outline:

(00:40) How RSP v3.0 Works

(19:05) You Came Here For An Argument

(21:27) The Problem Remains Unsolved

(25:22) Wow That Thing We Did Was Pretty Risky, Huh?

(26:18) Risk Report #1

(28:19) Listen All Yall Its Sabotage

(38:05) Looking Forward

(39:42) Claude Gov

(40:02) What Is A Strong Argument?

(41:12) Recursive Self-Improvement

(42:32) Non-Novel Chemical and Biological Weapons

(44:51) Novel Chemical and Biological Weapons

(45:39) Cross-Cutting Content (Section 6)

(48:48) Risk Report Report

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

Outline:

(01:41) Language Models Offer Mundane Utility

(03:00) Heads In The Sand

(07:05) Huh, Upgrades

(08:10) Mythos

(12:07) Whats In A Name

(14:59) On Your Marks

(16:10) Choose Your Fighter

(16:53) Get My Agent On The Line

(17:31) Deepfaketown and Botpocalypse Soon

(24:33) Cyber Lack Of Security

(29:08) Fun With Media Generation

(29:50) A Young Ladys Illustrated Primer

(30:53) They Took Our Jobs

(37:45) After They Take Our Jobs

(39:16) Gell-Mann Amnesia

(41:33) Get Involved

(43:25) In Other AI News

(46:41) Show Me the Money

(51:08) Quiet Speculations

(51:59) Explaining Persistent Model Parity

(55:37) Take a Moment

(01:00:54) OpenAI: The Histories

(01:06:04) The Department of AI War

(01:12:38) Department of AI Solidarity

(01:13:46) Writing For The AIs

(01:16:42) Quickly, Theres No Time

(01:16:46) The Quest for Sane Regulations

(01:18:10) Chip City

(01:20:07) You Received The Federal Framework

(01:21:02) The Week in Audio

(01:24:22) Rhetorical Innovation

(01:27:48) I Am The Very Human Of A Frontier Language Model

(01:38:01) Aligning a Smarter Than Human Intelligence is Difficult

(01:41:22) Aligning Fake Graphs Can Also Be Difficult

(01:49:32) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Anthropic has revised its Responsible Scaling Policy to v3.

The changes involved include abandoning many previous commitments, including one not to move ahead if doing so would be dangerous, citing that given competition they feel blindly following such a principle would not make the world safer.

Holden Karnofsky advocated for the changes. He maintains that the previous strategy of specific commitments was in error, and instead endorses the new strategy of having aspirational goals. He was not at Anthropic when the commitments were made.

My response to this will be two parts.

Today's post talks about considerations around Anthropic going back on its previous commitments, including asking to what extent Anthropic broke promises or benefited from people reacting to those promises, and how we should respond.

It is good, given that Anthropic was not going to keep its promises, that it came out and told us that this was the case, in advance. Thank you for that.

I still think that Anthropic importantly broke promises, that people relied upon, and did so in ways that made future trust and coordination, both with Anthropic and between labs and governments, harder. Admitting to the situation [...]

---

Outline:

(01:47) Promises, Promises

(03:10) Anthropic Responsible Scaling Policy v3

(03:32) That Could Have Gone Better

(04:36) Im Just Not Ready To Make a Commitment

(08:20) So Cold, So Alone

(12:24) Im Sorry I Gave You That Impression

(19:44) Fool Me Twice

(23:27) In My Defense I Was Left Unsupervised

(26:01) Drake Thomas Finds The Missing Mood

(28:49) Things That Could Have Been Brought To My Attention Yesterday (1)

(30:32) Things That Could Have Been Brought To My Attention Yesterday (2)

(36:13) What We Have Here Is A Failure To Communicate

(39:21) You Should See The Other Guy

(42:17) I Was Only Kidding

(43:12) They Cant Keep Getting Away With This

(44:07) Damn Your Sudden But Inevitable Betrayal

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Outline:

(03:37) Babies Are Awesome

(04:58) People Are Worried About AI Killing Everyone

(06:17) Freak Out

(06:47) Other People Are Not Worried About AI Killing Everyone

(09:27) Deepfaketown and Botpocalypse Soon

(10:15) Stopping The AI Race and A Narrow Path

(11:47) CEOs Know Their Roles

(13:28) The Call To Action

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Table of Contents

---

Outline:

(00:46) The OpenAI Foundation Exists

(07:55) Congress Exists

(10:47) China Self-Owns

(13:16) The Quest for Survival

(17:17) Alex Bores Watch

(21:15) You Received The Federal Framework

(26:55) Chip City

(29:16) Water Water Everywhere

(30:52) Senator Bernie Sanders Acts Authentically

(35:29) Pick Up The Phone

(35:54) Rhetorical Innovation

(46:03) Im A Conscious Robot

(50:21) How To Get Zvi To Read Your Paper

(54:18) People Really Hate AI

(54:34) Greetings From The Stop AI Protest

(01:01:47) Models Have Goals

(01:03:33) If I Was Two-Faced Would I Be Wearing This One

(01:06:25) Other People Are Not As Worried About AI Killing Everyone

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Last night, Anthropic was given its preliminary injunction, with a stay of seven days.

Emil Michael is a very angry person right now. So is the Honorable Judge Lin.

We were worried we would draw a judge that had no idea how any of this worked and would give the government absurd deference or buy into nonsense arguments.

That is not how it played out. Judge Lin very much understood the issues in play, as they did not require a technical background. She hammered the government in the hearing, and she wrote one of the most forceful, devastating judge opinions I have ever seen. It was an honor and sparked joy to be able to read it.

This post will proceed chronologically, picking up after the events of my last update.

If you want the short version and don’t care about the incremental steps, you can skip directly to Judge Lin Drops The Hammer, leaving the rest as a historical document and source for those who need to establish various facts going forward, including in court.

Logistical note: Due to breaking news, AI #161 Part 2 will be published on Monday. Then, if [...]

---

Outline:

(01:25) Anthropic Responds To The DoWs Brief

(12:05) Alan Rozenshtein and Others Suggest A Narrow Legal Way Out

(13:42) DoW Tried Another Uniquely Ill-Suited Theory Against Anthropic

(20:48) Other Views About The Situation From Back Then

(21:23) Potentially Long Suffering Judge Rita Lin Goes Hard At Hearing

(29:35) Emil Michael Tells On Himself

(41:54) Judge Lin Drops The Hammer

(49:57) Let Me Count The Ways

(53:35) Emil Michael Doubles Down Once Again

(55:57) What Happens Now

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The major technical advances this week were in agentic coding, as covered yesterday.

The major non-DoW political and alignment developments will be covered tomorrow.

The DoW vs. Anthropic trial continues. Judge Lin was very not happy with the government's case, which makes sense since the government has no case and was arguing a variety of Obvious Nonsense. The question now is how much preliminary relief Anthropic is entitled to. Assuming we find that out this week, I plan to cover that on Monday.

Beyond that, we have new iterations of questions we’ve dealt with time and again. The debate on jobs gets another cycle. Anthropic asked over 80,000 people what they think about AI, and has published those findings, nothing shocking but interesting throughout.

OpenAI is raising money again, although the terms raise some eyebrows. Elon Musk is announcing a grand chip project, but it was already kind of announced and it's not like we should believe him when he says such things.

I used this lull to drop a giant response to Open Socrates, which is technically a book review but uses that as a taking off point to outline a distinct philosophy [...]

---

Outline:

(01:44) Language Models Offer Mundane Utility

(02:44) Refine Your Paper

(04:57) Language Models Dont Offer Mundane Utility

(06:38) Huh, Upgrades

(06:47) On Your Marks

(10:53) Get My Agent On The Line

(12:42) Deepfaketown and Botpocalypse Soon

(15:07) Fun With Media Generation

(16:45) Greetings From The Torment Nexus

(17:05) A Young Ladys Illustrated Primer

(20:02) You Drive Me Crazy

(20:23) They Took Our Jobs

(31:24) They Are Hiring

(32:21) Levels of Friction

(33:28) In Other AI News

(34:48) Show Me the Money

(43:34) Quickly, Theres No Time

(44:30) The Week in Audio

(46:46) 80,000 Interviews About AI

(52:55) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.