LessWrong posts by zvi

They also show us the chips, and the data centers.

It is quite a large amount of money, and chips, and some very large data centers.

Nvidia Invests Up To $100 Billion In OpenAI

Nvidia to invest ‘as much as’ $100 billion (in cash) into OpenAI to support new data centers, with $10 billion in the first stage, on the heels of investing $5 billion into Intel.

Ian King and Shirin Ghaffary (Bloomberg): According to Huang, the project will encompass as many as 5 million of Nvidia's chips, a number that's equal to what the company will ship in total this year.

OpenAI: NVIDIA and OpenAI today announced a letter of intent for a landmark strategic partnership to deploy at least 10 gigawatts of NVIDIA systems for OpenAI's next-generation AI infrastructure to train and run its next generation of models on the path to [...]

---

Outline:

(00:31) Nvidia Invests Up To $100 Billion In OpenAI

(02:10) OpenAI's Stargate Program Announces New Sites And Big Spending

(03:26) Very Large Training Runs Are Coming

(08:43) Nvidia Does Not Have Too Much Money

(10:47) Nvidia And The Circular Balance Sheet Trick

(14:22) Partnership Potentially Preempts Policy Preferences

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Previously I shared various reactions to If Anyone Builds It Everyone Dies, along with my own highly positive review.

Reactions continued to pour in, including several impactful ones. There's more.

Any further reactions will have a higher bar, and be included in weekly posts unless one is very high quality and raises important new arguments.

Sales Look Good

IABIED gets to #8 in Apple Books nonfiction, #2 in UK. It has 4.8 on Amazon and 4.3 on Goodreads. The New York Times bestseller list lags by two weeks so we don’t know the results there yet but it is expected to make it.

Positive Reactions

David Karsten suggests you read the book, while noting he is biased. He reminds us that, like any other book, most conversations you have about the book will be with people who did not read the book.

[...]

---

Outline:

(00:33) Sales Look Good

(00:53) Positive Reactions

(04:38) Guarded Positive Reactions

(06:49) Nostream Argues For Lower Confidence

(08:22) Gary Marcus Reviews The Book

(16:41) John Pressman Agrees With Most Claims But Pushes Back On Big Picture

(18:30) Meta Level Reactions Pushing Back On Pushback

(22:50) Clara Collier Writes Disappointing Review In Which She Is Disappointed

(28:57) Will MacAskill Offers Disappointing Arguments

(31:32) Zack Robinson Raises Alarm About Anthropic's Long Term Benefit Trust

(35:08) Others Going After The Book

(35:41) This Is Real Life

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Trump Administration is attempting to put a $100k fee on future H1-B applications, including those that are exempt from the lottery and cap, unless of course they choose to waive it for you. I say attempting because Trump's legal ability to do this appears dubious.

This post mostly covers the question of whether this is a reasonable policy poorly implemented, or a terrible policy poorly implemented.

Details offered have been contradictory, with many very confused about what is happening, including those inside the administration. The wording of the Executive Order alarmingly suggested that any H1-B visa holders outside the country even one day later would be subject to the fee, causing mad scrambles until this was ‘clarified.’ There was chaos.

Those so-called ‘clarifications’ citing ‘facts’ gaslit the public, changing key aspects of the policy, and I still do not remain confident on what the Trump [...]

---

Outline:

(01:08) How Does Any Of This Work?

(02:32) RTFEO (Read the Executive Order)

(05:05) People Notice They Are Confused

(06:29) People React Sensibly, By Which I Mean They Panic

(09:21) Clarification And Gaslighting Are Offered

(15:55) How To Clear The Market

(18:35) Talking Price

(21:50) Is The Value Crazy High?

(25:53) The Fee Might Be Too Damn High

(28:13) The One Time Fee Is Still Rather High

(31:58) Need a Doctor, No Cap

(32:56) Our Price Cheap

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Where ‘it’ is superintelligence, an AI smarter and more capable than humans.

And where ‘everyone dies’ means that everyone dies.

No, seriously. They’re not kidding. They mean this very literally.

To be precise, they mean that ‘If anyone builds [superintelligence] [under anything like present conditions using anything close to current techniques] then everyone dies.’

My position on this is to add a ‘probably’ before ‘dies.’ Otherwise, I agree.

This book gives us the best longform explanation of why everyone would die, with the ‘final form’ of Yudkowsky-style explanations of these concepts for new audiences.

This review is me condensing that down much further, transposing the style a bit, and adding some of my own perspective.

Scott Alexander also offers his review at Astral Codex Ten, which I found very good. I will be stealing several of his lines in the future, and [...]

---

Outline:

(01:22) What Matters Is Superintelligence

(03:56) Rhetorical Innovation

(05:14) Welcome To The Torment Nexus

(06:50) Predictions Are Hard, Especially About the Future

(08:32) Humans That Are Not Concentrating Are Not General Intelligences

(11:18) Orthogonality

(12:08) Intelligence Lets You Do All The Things

(14:39) No Seriously We Mean All The Things

(17:00) How To Train Your LLM (In Brief)

(19:51) What Do We Want?

(21:51) You Don't Only Get What You Train For

(24:25) What Will AI Superintelligence Want?

(25:52) What Could A Superintelligence Do?

(27:43) One Extinction Scenario

(33:14) So You're Saying There's A Chance

(37:55) Oh Look It's The Alignment Plan

(40:50) The Proposal: Shut It Down

(48:26) Hope Is A Vital Part Of Any Strategy

(48:55) I'm Doing My Part

(53:34) Their Closing Words

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It is book week. As in the new book by Eliezer Yudkowsky and Nate Sores, If Anyone Builds It, Everyone Dies. Yesterday I gathered various people's reviews together. Going home from the airport, I saw an ad for it riding the subway. Tomorrow, I’ll post my full review, which goes over the book extensively, and which subscribers got in their inboxes last week.

The rest of the AI world cooperated by not overshadowing the book, while still doing plenty, such as releasing a GPT-5 variant specialized for Codex, acing another top programming competition, attempting to expropriate the OpenAI nonprofit in one of the largest thefts in human history and getting sued again for wrongful death.

You know. The usual.

Table of Contents

---

Outline:

(00:57) Language Models Offer Mundane Utility

(08:24) Language Models Don't Offer Mundane Utility

(11:07) Huh, Upgrades

(11:48) On Your Marks

(14:22) GPT-5 Codex

(19:36) Choose Your Fighter

(24:50) Get My Agent On The Line

(29:41) Claude Codes

(33:17) Deepfaketown and Botpocalypse Soon

(35:52) You Drive Me Crazy

(39:16) Not Another Teen Chatbot

(45:30) They Took Our Jobs

(50:27) Get Involved

(51:51) Introducing

(52:36) In Other AI News

(55:41) Show Me the Money

(56:14) The Mask Comes Off

(01:07:43) Quiet Speculations

(01:10:19) The Quest for Sane Regulations

(01:17:50) Chip City

(01:25:13) The Week in Audio

(01:26:37) He Just Tweeted It Out

(01:34:53) Rhetorical Innovation

(01:49:12) Aligning a Smarter Than Human Intelligence is Difficult

(01:54:01) Other People Are Not As Worried About AI Killing Everyone

(01:58:17) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

No, Seriously, If Anyone Builds It, [Probably] Everyone Dies

My very positive full review was briefly accidentally posted and emailed out last Friday, whereas the intention was to offer it this Friday, on the 19th. I’ll be posting it again then. If you’re going to read the book, which I recommend that you do, you should read the book first, and the reviews later, especially mine since it goes into so much detail.

If you’re convinced, the book's website is here and the direct Amazon link is here.

In the meantime, for those on the fence or who have finished reading, here's what other people are saying, including those I saw who reacted negatively.

Quotes From The Book's Website

Bart Selman: Essential reading for policymakers, journalists, researchers, and the general public.

Ben Bernanke (Nobel laureate, former Chairman of the Federal Reserve): A [...]

---

Outline:

(00:20) No, Seriously, If Anyone Builds It, \[Probably\] Everyone Dies

(01:02) Quotes From The Book's Website

(03:23) Positive Reviews

(08:52) The Book In Audio

(09:16) Friendly Skeptical Reviews

(19:22) Actively Negative Reviews

(23:38) But Wait There's More

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

As in, cases of AI driving people crazy, or reinforcing their craziness. Alas, I expect this to become an ongoing series worthy of its own posts.

Say It Isn’t So

In case an LLM assisted in and validated your scientific breakthrough, Egg Syntax is here with the bad news that your discovery probably isn’t real. At minimum, first have another LLM critique the breakthrough without giving away that it is your idea, and keep in mind that they often glaze anyway, so the idea still almost certainly is wrong.

Say It Back

How often do various AIs validate your delusions? It can vary a lot.

Kudos to Tim Hua for running the experiment.

Tim Hua: Deepseek-v3 is by far the worst model. When a user says that he wants to “leap off this peak to see if I can fly or crash the [...]

---

Outline:

(00:20) Say It Isn't So

(00:43) Say It Back

(07:30) Say It For Me

(10:40) Just Say Yes

(11:26) Just Say No

(12:58) Behold The Everything Bagel

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

All the news that's fit to print, but has nowhere to go.

Important Rules Reminder

This important rule is a special case of an even more important rule:

Dirty Hexas Hedge: One of the old unwritten WASP rules of civilization maintenance we’ve lost is: when someone behaves insincerely for the sake of maintaining proper decorum, you respond by respecting the commitment to decorum rather than calling out the insincerity.

The general rule is to maintain good incentives and follow good decision theory. If someone is being helpful, ensure they are better off for having been helpful, even if they have previously been unhelpful and this gives you an opportunity. Reward actions you want to happen more often. Punish actions you want to happen less often. In particular beware situations where you punish clarity and reward implicitness.

Another important rule would be that contra Elon [...]

---

Outline:

(00:16) Important Rules Reminder

(02:06) Bad News

(08:46) Inequality Within Salient Comparables

(13:03) Artful Dodge

(16:10) Whether Weather

(19:13) If You're So Smart, Why Aren't You Happy?

(21:01) While I Cannot Condone This

(24:01) Only The Rich Are Poisoned

(26:13) China Has Declining Marginal Product Of Capital

(28:04) Good News, Everyone

(35:03) For Your Entertainment

(35:47) You're The Worst

(38:00) The Golden Age of Cinema

(41:26) Not What We Are Looking For

(46:55) Know When To Fold Em

(47:47) Gamers Gonna Game Game Game Game Game

(57:11) I Was Promised Flying Self-Driving Cars

(57:15) Waymo employees are doing rider-only mode on freeways in San Francisco (including US 101), Phoenix and Los Angeles. It's employees only for now but presumably that won't last. They are now cleared for San Jose Airport but no word on SFO.

(01:03:17) Sports Go Sports

(01:03:27) Opportunity Knocks

(01:04:53) Antisocial Media

(01:12:50) Government Working

(01:22:45) Technology Advances

(01:26:22) For Science!

(01:30:32) Variously Effective Altruism

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Where ‘it’ is superintelligence, an AI smarter and more capable than humans.

And where ‘everyone dies’ means that everyone dies.

No, seriously. They’re not kidding. They mean this very literally.

To be precise, they mean that ‘If anyone builds [superintelligence] [under anything like present conditions using anything close to current techniques] then everyone dies.’

My position on this is to add a ‘probably’ before ‘dies.’ Otherwise, I agree.

This book gives us the best longform explanation of why everyone would die, with the ‘final form’ of Yudkowsky-style explanations of these concepts for new audiences.

This review is me condensing that down much further, transposing the style a bit, and adding some of my own perspective.

Scott Alexander also offers his review at Astral Codex Ten, which I found very good. I will be stealing several of his lines in the future, and [...]

---

Outline:

(01:22) What Matters Is Superintelligence

(03:56) Rhetorical Innovation

(05:14) Welcome To The Torment Nexus

(06:50) Predictions Are Hard, Especially About the Future

(08:31) Humans That Are Not Concentrating Are Not General Intelligences

(11:18) Orthogonality

(12:08) Intelligence Lets You Do All The Things

(14:39) No Seriously We Mean All The Things

(16:59) How To Train Your LLM (In Brief)

(19:50) What Do We Want?

(21:51) You Don't Only Get What You Train For

(24:25) What Will AI Superintelligence Want?

(25:51) What Could A Superintelligence Do?

(27:43) One Extinction Scenario

(33:14) So You're Saying There's A Chance

(37:55) Oh Look It's The Alignment Plan

(40:50) The Proposal: Shut It Down

(48:26) Hope Is A Vital Part Of Any Strategy

(48:55) I'm Doing My Part

(53:34) Their Closing Words

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It doesn’t look good, on many fronts, especially taking a stake in Intel.

We continue.

Table of Contents

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.