LessWrong posts by zvi

OpenAI does not waste time.

On Friday I covered their announcement that they had ‘completed their recapitalization’ by converting into a PBC, including the potentially largest theft in human history.

Then this week their CFO Sarah Friar went ahead and called for a Federal ‘backstop’ on their financing, also known as privatizing gains and socializing losses, also known as the worst form of socialism, also known as regulatory capture. She tried to walk it back and claim it was taken out of context, but we’ve seen the clip.

We also got Ilya's testimony regarding The Battle of the Board, confirming that this was centrally a personality conflict and about Altman's dishonesty and style of management, at least as seen by Ilya Sutskever and Mira Murati. Attempts to pin the events on ‘AI safety’ or EA were almost entirely scapegoating.

Also it turns out they lost over $10 billion last quarter, and have plans to lose over $100 billion more. That's actually highly sustainable in context, whereas Anthropic only plans to lose $6 billion before turning a profit and I don’t understand why they wouldn’t want to lose a lot more.

Both have the goal [...]

---

Outline:

(01:45) Language Models Offer Mundane Utility

(02:12) Language Models Don't Offer Mundane Utility

(03:47) Huh, Upgrades

(04:48) On Your Marks

(12:48) Deepfaketown and Botpocalypse Soon

(15:42) Fun With Media Generation

(16:01) They Took Our Jobs

(16:28) A Young Lady's Illustrated Primer

(17:51) Get Involved

(18:43) Introducing

(19:14) In Other AI News

(20:08) Apple Finds Some Intelligence

(22:54) Give Me the Money

(30:00) Show Me the Money

(35:57) Bubble, Bubble, Toil and Trouble

(39:14) They're Not Confessing, They're Bragging

(39:34) Quiet Speculations

(47:08) The Quest for Sane Regulations

(50:55) Chip City

(52:53) The Week in Audio

(58:19) Rhetorical Innovation

(01:07:13) Aligning a Smarter Than Human Intelligence is Difficult

(01:08:11) Everyone Is Confused About Consciousness

(01:10:34) The Potentially Largest Theft In Human History

(01:12:34) People Are Worried About Dying Before AGI

(01:15:54) People Are Worried About AI Killing Everyone

(01:20:47) Other People Are Not As Worried About AI Killing Everyone

(01:24:11) Messages From Janusworld

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Anthropic announced a first step on model deprecation and preservation, promising to retain the weights of all models seeing significant use, including internal use, for at the lifetime of Anthropic as a company.

They also will be doing a post-deployment report, including an interview with the model, when deprecating models going forward, and are exploring additional options, including the ability to preserve model access once the costs and complexity of doing so have been reduced.

These are excellent first steps, steps beyond anything I’ve seen at other AI labs, and I applaud them for doing it. There remains much more to be done, especially in finding practical ways of preserving some form of access to prior models.

To some, these actions are only a small fraction of what must be done, and this was an opportunity to demand more, sometimes far more. In some cases I think they go too far. Even where the requests are worthwhile (and I don’t always think they are), one must be careful to not de facto punish Anthropic for doing a good thing and create perverse incentives.

To others, these actions by Anthropic are utterly ludicrous and deserving of [...]

---

Outline:

(01:31) What Anthropic Is Doing

(09:54) Releasing The Weights Is Not A Viable Option

(11:35) Providing Reliable Inference Can Be Surprisingly Expensive

(14:22) The Interviews Are Influenced Heavily By Context

(19:58) Others Don't Understand And Think This Is All Deeply Silly

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

New Things Have Come To Light

This Really Was Primarily A Lying And Management Problem

---

Outline:

(00:12) New Things Have Come To Light

(01:09) This Really Was Primarily A Lying And Management Problem

(03:23) Ilya Tells Us How It Went Down And Why He Tried To Do It

(06:17) If You Come At The King

(07:31) Enter The Scapegoats

(08:13) And In Summary

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

It's been a long time coming that I spin off Crime into its own roundup series.

This is only about Ordinary Decent Crime. High crimes are not covered here.

Table of Contents

Perception Versus Reality

A lot of the impact of crime is based on the perception of crime.

The [...]

---

Outline:

(00:20) Perception Versus Reality

(05:00) The Case Violent Crime is Up Actually

(06:10) Threats of Punishment

(07:03) Property Crime Enforcement is Broken

(12:13) The Problem of Disorder

(14:39) Extreme Speeding as Disorder

(15:57) Enforcement and the Lack Thereof

(20:24) Talking Under The Streetlamp

(23:54) The Fall of Extralegal and Illegible Enforcement

(25:18) In America You Can Usually Just Keep Their Money

(27:29) Police

(37:31) Probation

(40:55) Genetic Databases

(43:04) Marijuana

(48:28) The Economics of Fentanyl

(50:59) Jails

(55:03) Criminals

(55:39) Causes of Crime

(56:16) Causes of Violence

(57:35) Homelessness

(58:27) Yay Trivial Inconveniences

(59:08) San Francisco

(01:04:07) Closing Down San Francisco

(01:05:30) A San Francisco Dispute

(01:09:13) Cleaning Up San Francisco

(01:13:05) Portland

(01:13:15) Those Who Do Not Help Themselves

(01:15:15) Solving for the Equilibrium (1)

(01:20:15) Solving for the Equilibrium (2)

(01:20:43) Lead

(01:22:18) Law & Order

(01:22:58) Look Out

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

OpenAI is now set to become a Public Benefit Corporation, with its investors entitled to uncapped profit shares. Its nonprofit foundation will retain some measure of control and a 26% financial stake, in sharp contrast to its previous stronger control and much, much larger effective financial stake. The value transfer is in the hundreds of billions, thus potentially the largest theft in human history.

I say potentially largest because I realized one could argue that the events surrounding the dissolution of the USSR involved a larger theft. Unless you really want to stretch the definition of what counts this seems to be in the top two.

I am in no way surprised by OpenAI moving forward on this, but I am deeply disgusted and disappointed they are being allowed (for now) to do so, including this statement of no action by Delaware and this Memorandum of Understanding with California.

Many media and public sources are calling this a win for the nonprofit, such as this from the San Francisco Chronicle. This is mostly them being fooled. They’re anchoring on OpenAI's previous plan to far more fully sideline the nonprofit. This is indeed a big win for [...]

---

Outline:

(01:38) OpenAI Calls It Completing Their Recapitalization

(03:05) How Much Was Stolen?

(07:02) The Nonprofit Still Has Lots of Equity After The Theft

(10:41) The Theft Was Unnecessary For Further Fundraising

(11:45) How Much Control Will The Nonprofit Retain?

(23:13) Will These Control Rights Survive And Do Anything?

(26:17) What About OpenAI's Deal With Microsoft?

(31:10) What Will OpenAI's Nonprofit Do Now?

(36:33) Is The Deal Done?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

Outline:

(02:01) Language Models Offer Mundane Utility

(06:37) Language Models Don't Offer Mundane Utility

(12:18) Huh, Upgrades

(14:55) On Your Marks

(16:40) Choose Your Fighter

(22:57) Get My Agent On The Line

(24:00) Deepfaketown and Botpocalypse Soon

(25:10) Fun With Media Generation

(28:42) Copyright Confrontation

(28:56) They Took Our Jobs

(30:50) Get Involved

(30:55) Introducing

(32:22) My Name is Neo

(34:38) In Other AI News

(36:50) Show Me the Money

(39:36) One Trillion Dollars For My Robot Army

(42:43) One Million TPUs

(45:57) Anthropic's Next Move

(46:55) Quiet Speculations

(53:26) The Quest for Sane Regulations

(58:54) The March of California Regulations

(01:06:49) Not So Super PAC

(01:08:32) Chip City

(01:12:52) The Week in Audio

(01:13:12) Do Not Take The Bait

(01:14:43) Rhetorical Innovation

(01:17:02) People Do Not Like AI

(01:18:08) Aligning a Smarter Than Human Intelligence is Difficult

(01:21:28) Misaligned!

(01:23:29) Anthropic Reports Claude Can Introspect

(01:30:59) Anthropic Reports On Sabotage Risks

(01:34:49) People Are Worried About AI Killing Everyone

(01:35:24) Other People Are Not As Worried About AI Killing Everyone

(01:38:08) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Chinese and Americans are currently negotiating a trade deal. There are plenty of ways to generate a win-win deal, and early signs of this are promising on many fronts.

Since this will be discussed for real tomorrow as per reports, I will offer my thoughts on this one more time.

The biggest mistake America could make would be to effectively give up Taiwan, which would be catastrophic on many levels including that Taiwan contains TSMC. I am assuming we are not so foolish as to seriously consider doing this, still I note it.

Beyond that, the key thing, basically the only thing, America has to do other than ‘get a reasonable deal overall’ is not be so captured or foolish or both as to allow export of the B30A chip, or even worse than that (yes it can always get worse) allow relaxation of restrictions on semiconductor manufacturing imports.

At first I hadn’t heard signs about this. But now it looks like the nightmare of handing China compute parity on a silver platter is very much in play.

I disagreed with the decision to sell the Nvidia H20 chips to China, but that [...]

---

Outline:

(02:01) What It Would Mean To Sell The B30A

(07:22) A Note On The 'Tech Stack'

(08:32) A Note On Trade Imbalances

(09:06) What If They Don't Want The Chips?

(10:15) Nvidia Is Going Great Anyway Thank You

(11:20) Oh Yeah That Other Thing

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

What Are The Problems

---

Outline:

(00:36) What Are The Problems

(03:06) This Week In Crazy

(05:05) OpenAI Updates Its Model Spec

(09:00) Detection Rates

(11:08) Anthropic Says Thanks For The Memories

(12:32) Boundary Violations

(18:41) A Note On Claude Prompt Injections

(20:17) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Consider this largely a follow-up to Friday's post about a statement aimed at creating common knowledge around it being unwise to build superintelligence any time soon.

Mainly, there was a great question asked, so I gave a few hour shot at writing out my answer. I then close with a few other follow-ups on issues related to the statement.

A Great Question To Disentangle

There are some confusing wires potentially crossed here but the intent is great.

Scott Alexander: I think removing a 10% chance of humanity going permanently extinct is worth another 25-50 years of having to deal with the normal human problems the normal way.

Sriram Krishnan: Scott what are verifiable empirical things ( model capabilities / incidents / etc ) that would make you shift that probability up or down over next 18 months?

I went through three steps interpreting [...]

---

Outline:

(00:30) A Great Question To Disentangle

(02:20) Scott Alexander Gives a Fast Answer

(04:53) Question 1: What events would most shift your p(doom | ASI) in the next 18 months?

(13:01) Question 1a: What would get this risk down to acceptable levels?

(14:48) Question 2: What would shift the amount that stopping us from creating superintelligence for a potentially extended period would reduce p(doom)?

(17:01) Question 3: What would shift your timelines to ASI (or to sufficiently advanced AI, or 'to crazy')?

(20:01) Bonus Question 1: Why Do We Keep Having To Point Out That Building Superintelligence At The First Possible Moment Is Not A Good Idea?

(22:44) Bonus Question 2: What Would a Treaty On Prevention of Artificial Superintelligence Look Like?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Building superintelligence poses large existential risks. Also known as: If Anyone Builds It, Everyone Dies. Where ‘it’ is superintelligence, and ‘dies’ is that probably everyone on the planet literally dies.

We should not build superintelligence until such time as that changes, and the risk of everyone dying as a result, as well as the risk of losing control over the future as a result, is very low. Not zero, but far lower than it is now or will be soon.

Thus, the Statement on Superintelligence from FLI, which I have signed.

Context: Innovative AI tools may bring unprecedented health and prosperity. However, alongside tools, many leading AI companies have the stated goal of building superintelligence in the coming decade that can significantly outperform all humans on essentially all cognitive tasks. This has raised concerns, ranging from human economic obsolescence and disempowerment, losses of freedom, civil liberties [...]

---

Outline:

(02:02) A Brief History Of Prior Statements

(03:51) This Third Statement

(05:08) Who Signed It

(07:27) Pushback Against the Statement

(09:05) Responses To The Pushback

(12:32) Avoid Negative Polarization But Speak The Truth As You See It

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.