LessWrong posts by zvi

Previously: The Fundamentals, The Gamblers, The Business

We have now arrived at the topics most central to this book, aka ‘The Future.’

Rationalism and Effective Altruism (EA)

The Manifest conference was also one of the last reporting trips that I made for this book. And it confirmed for me that the River is real—not just some literary device I invented. (6706)

Yep. The River is real.

I consider myself, among many things, a straight up rationalist.

I do not consider myself an EA, and never have.

This completes the four quadrants of the two-by-two of [does Nate knows it well, does Zvi knows it well]. The first two, where Nate was in his element, went very well. The third clearly was less exacting, as one would expect, but pretty good.

Now I have the information advantage, even more than I did [...]

---

Outline:

(00:16) Rationalism and Effective Altruism (EA)

(06:01) Cost-Benefit Analysis

(09:04) How About Trying At All

(10:11) The Virtues of Rationality

(11:56) Effective Altruism and Rationality, Very Different of Course

(24:37) The Story of OpenAI

(30:19) Altman, OpenAI and AI Existential Risk

(38:26) Tonight at 11: Doom

(01:00:39) AI Existential Risk: They’re For It

(01:07:42) To Pause or Not to Pause

(01:11:11) You Need Better Decision Theory

(01:15:27) Understanding the AI

(01:19:43) Aligning the AI

(01:23:50) A Glimpse of Our Possible Future

(01:28:16) The Closing Motto

---

First published:

Source:

Narrated by TYPE III AUDIO.

We interrupt Nate Silver week here at Don’t Worry About the Vase to bring you some rather big AI news: OpenAI and Sam Altman are planning on fully taking their masks off, discarding the nonprofit board's nominal control and transitioning to a for-profit B-corporation, in which Sam Altman will have equity.

We now know who they are and have chosen to be. We know what they believe in. We know what their promises and legal commitments are worth. We know what they plan to do, if we do not stop them.

They have made all this perfectly clear. I appreciate the clarity.

On the same day, Mira Murati, the only remaining person at OpenAI who in any visible way opposed Altman during the events of last November, resigned without warning along with two other senior people, joining a list that now includes among others several OpenAI [...]

---

Outline:

(01:51) Language Models Offer Mundane Utility

(04:17) Language Models Don’t Offer Mundane Utility

(06:35) The Mask Comes Off

(18:51) Fun with Image Generation

(18:54) Deepfaketown and Botpocalypse Soon

(19:50) They Took Our Jobs

(20:49) The Art of the Jailbreak

(21:28) OpenAI Advanced Voice Mode

(26:03) Introducing

(28:29) In Other AI News

(30:30) Quiet Speculations

(34:00) The Quest for Sane Regulations

(42:21) The Week in Audio

(42:47) Rhetorical Innovation

(56:50) Aligning a Smarter Than Human Intelligence is Difficult

(01:00:36) Other People Are Not As Worried About AI Killing Everyone

(01:01:53) The Lighter Side

The original text contained 6 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Previously: The Fundamentals, The Gamblers

Having previously handled the literal gamblers, we are ready to move on to those who Do Business using Riverian principles.

Or at least while claiming to use Riverian principles, since Silicon Valley doesn’t fit into the schema as cleanly as many other groups. That's where we begin this section, starting at the highest possible conceptual level.

Time to talk real money.

Why Can You Do This Trade?

First law of trading: For you to buy, someone must sell. Or for you to sell, someone must buy. And there can’t be someone else doing the trade before you did it.

Why did they do that, and why did no one else take the trade first? Until you understand why you are able to do this trade, you should be highly suspicious.

“Every single thing we do, I can [...]

---

Outline:

(00:41) Why Can You Do This Trade?

(03:08) In a World of Venture Capital

(10:54) Short Termism Hypothesis

(12:42) Non-Determinism and its Discontents

(14:57) The Founder, the Fox and the Hedgehog

(17:11) The Team to Beat

(24:22) Silicon Valley Versus Risk

(35:14) The Keynesian Beauty Contest

(40:57) The Secret of Their Success is Deal Flow

(50:00) The Valley Beside the River

(53:07) Checkpoint Three

(53:37) Fun With SBF and Crypto Fraud

(01:01:53) Other Crypto Thoughts Unrelated to SBF

(01:04:50) Checkpoint Four

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Previously: Book Review: On the Edge: The Fundamentals

As I said in the Introduction, I loved this part of the book. Let's get to it.

Poker and Game Theory

When people talk about game theory, they mostly talk solving for the equilibrium, and how to play your best game or strategy (there need not be a formal game) against adversaries who are doing the same.

I think of game theory like Frank Sinatra thinks of New York City: “If I can make it there, I’ll make it anywhere.” If you can compete against people performing at their best, you’re going to be a winner in almost any game you play. But if you build a strategy around exploiting inferior competition, it's unlikely to be a winning approach outside of a specific, narrow setting. What plays well in Peoria doesn’t necessarily play well in New York. [...]

---

Outline:

(00:18) Poker and Game Theory

(06:53) Sports Randomized Sports

(11:17) Knowing Theory Versus Memorization Versus Practice

(16:15) More About Tells

(19:20) Feeling the Probabilities

(20:35) Feeling Sad About It

(28:33) The Iowa Gambling Task

(31:39) The Greatest Risk

(37:20) Tournament Poker Is Super High Variance

(42:42) The Art of the Degen

(48:43) Why Do They Insist on Calling it Luck

(51:56) The Poker Gender Gap

(54:36) A Potential Cheater

(58:30) Making a Close Decision

(01:00:19) Other Games at the Casino

(01:03:22) Slot Machines Considered Harmful

(01:08:23) Where I Draw the Line

(01:11:14) A Brief History of Vegas and Casinos (as told by Nate Silver)

(01:16:44) We Got Us a Whale

(01:21:41) Donald Trump and Atlantic City Were Bad At Casinos

(01:25:17) How To Design a Casino

(01:26:46) The Wide World of Winning at Sports Gambling

(01:41:01) Limatime

(01:43:45) The Art of Getting Down

(01:45:29) Oh Yeah That Guy

(01:55:34) The House Sometimes Wins

(02:01:24) The House Is Probably a Coward

(02:11:19) DFS and The Problem of Winners

(02:16:08) Balancing the Action

(02:18:44) The Market Maker

(02:22:45) The Closing Line is Hard to Beat

(02:25:11) Winning is Hard

(02:29:58) What Could Be, Unburdened By What Has Been

(02:34:52) Finding Edges Big and Small

(02:40:12) Checkpoint Two

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The most likely person to write On the Edge was Nate Silver.

Grok thinks the next most likely was Michael Lewis, followed by a number of other writers of popular books regarding people thinking different.

I see why Grok would say that, but it is wrong.

The next most likely person was Zvi Mowshowitz.

I haven’t written a book for this type of audience, a kind of smarter business-book, but that seems eminently within my potential range.

On the Edge is a book about those living On The Edge, the collection of people who take risk and think probabilistically and about expected value. It centrally covers poker, sports betting, casinos, Silicon Valley, venture capital, Sam Bankman-Fried, effective altruism, AI and existential risk.

Collectively, Nate Silver calls this cultural orientation The River.

It is contrasted with The Village, which comprises roughly the mainstream [...]

---

Outline:

(02:53) Overview

(07:56) Introduction: The River

(14:21) Nate Silver Comes Home to The River

(18:29) Nate (In General) Makes One Critical Mistake

(18:57) The Village Idiots

(22:22) Alone in the Wilderness

(25:46) Why the River Hates the Village

(29:57) Nate Silver's History of River Versus Village

(38:46) Spending Time at Airports

(41:14) The Coin Flip

(42:15) The Other Risk Takers

(49:45) Aside on Covid

(50:46) What is a Contrarian?

(53:49) Prediction Market Smackdown

(56:53) Checkpoint One

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I’d split the latest housing roundup into local versus global questions. I was planning on waiting a bit between them.

Then Joe Biden decided to propose a version of the worst possible thing.

So I guess here we are.

What is the organizing principle of Bidenomics?

Restrict Supply and Subsidize Demand (1)

This was the old counterproductive Biden proposal:

Unusual Whales: Biden to propose $5,000 credit for first-time home buyers, per WaPo.

The Rich: House prices about to go up $5,000 everywhere.

Under current conditions this is almost a pure regressive tax, a transfer from those too poor to own a home to those who can afford to buy one, or who previously owned one and no longer do.

If there were no restrictions on the supply of housing, such that the price of a house equalled the cost of [...]

---

Outline:

(00:24) Restrict Supply and Subsidize Demand (1)

(08:55) Restrict Supply and Subsidize Demand (2): Rent Control

(16:57) You Should See the Other Guy

(21:18) Stop Restricting Supply

(23:21) All Supply is Good Supply

(26:53) ‘Inclusionary’ Zoning

(28:37) The Worst Take

(33:27) Where and With Whom People Want To Live

(36:44) Matching

(39:58) Universality

(41:42) The Value of Land

(43:52) The Doom Loop

(51:05) How Are Sale Prices So Out of Whack with Rents and Income?

(52:18) Questioning Superstar Status

(54:47) Window Shopping

(58:18) Minimum Viable Product

(01:00:53) Construction Costs

(01:02:57) Elevator Action

(01:11:09) Housing Theory of Everything

(01:13:07) Zoning By Prohibitive Permit

(01:14:55) YIGBY?

(01:15:24) The True NIMBY

(01:16:45) The Definition of Chutzpah

(01:18:23) In Other Housing News

(01:18:53) Rhetoric

(01:21:33) Environmentalists Should Favor Density

(01:23:03) Do Not Give the People What They Want

(01:26:05) Housing Construction in the UK

(01:27:25) The Funniest Possible Thing

(01:29:07) Other Funny Things

The original text contained 35 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The big news of the week was of course OpenAI releasing their new model o1. If you read one post this week, read that one. Everything else is a relative sideshow.

Meanwhile, we await Newsom's decision on SB 1047. The smart money was always that Gavin Newsom would make us wait before offering his verdict on SB 1047. It's a big decision. Don’t rush him. In the meantime, what hints he has offered suggest he's buying into some of the anti-1047 talking points. I’m offering a letter to him here based on his comments, if you have any way to help convince him now would be the time to use that. But mostly, it's up to him now.

Table of Contents

---

Outline:

(00:49) Language Models Offer Mundane Utility

(02:11) Language Models Don’t Offer Mundane Utility

(03:34) Deepfaketown and Botpocalypse Soon

(05:59) They Took Our Jobs

(07:34) Get Involved

(08:09) Introducing

(10:11) In Other AI News

(13:20) Quiet Speculations

(15:15) Intelligent Design

(19:15) SB 1047: The Governor Ponders

(27:13) Letter to Newsom

(31:36) The Quest for Sane Regulations

(34:19) Rhetorical Innovation

(42:13) Claude Writes Short Stories

(45:54) Questions of Sentience

(48:22) People Are Worried About AI Killing Everyone

(49:56) The Lighter Side

The original text contained 6 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's that time again for all the sufficiently interesting news that isn’t otherwise fit to print, also known as the Monthly Roundup.

Bad News

Beware the failure mode in strategy and decisions that implicitly assumes competence, or wishes away difficulties, and remember to reverse all advice you hear.

Stefan Schubert (quoting Tyler Cowen on raising people's ambitions often being very high value): I think lowering others’ aspirations can also be high-return. I know of people who would have had a better life by now if someone could have persuaded them to pursue more realistic plans.

Rob Miles: There's a specific failure mode which I don’t have a name for, which is similar to “be too ambitious” but is closer to “have an unrealistic plan”. The illustrative example I use is:

Suppose by some strange circumstance you have to represent your country at olympic gymnastics [...]

---

Outline:

(00:14) Bad News

(03:45) Anti-Social Media

(07:20) Technology Advances

(08:56) High Seas Piracy is Bad

(12:17) The Michelin Curse

(15:21) What's the Rush?

(17:48) Good News, Everyone

(19:49) Let it Go

(22:09) Yay Air Conditioning

(23:27) Beast of a Memo

(36:58) For Science!

(37:18) For Your Entertainment

(39:47) Properly Rated

(45:26) Government Working

(49:21) Grapefruit Diet

(58:38) Gamers Gonna Game Game Game Game Game

(01:06:09) Gamers Winning At Life

(01:10:52) I Was Promised Flying Self-Driving Cars

(01:11:21) While I Cannot Condone This

(01:17:21) Nostalgia

(01:23:51) The Lighter Side

The original text contained 12 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Terrible name (with a terrible reason, that this ‘resets the counter’ on AI capability to 1, and ‘o’ as in OpenAI when they previously used o for Omni, very confusing). Impressive new capabilities in many ways. Less impressive in many others, at least relative to its hype.

Clearly this is an important capabilities improvement. However, it is not a 5-level model, and in important senses the ‘raw G’ underlying the system hasn’t improved.

GPT-o1 seems to get its new capabilities by taking (effectively) GPT-4o, and then using extensive Chain of Thought (CoT) and quite a lot of tokens. Thus that unlocks (a lot of) what that can unlock. We did not previously know how to usefully do that. Now we do. It gets much better at formal logic and reasoning, things in the ‘system 2’ bucket. That matters a lot for many tasks, if not as much [...]

---

Outline:

(01:26) Introducing GPT-o1

(05:05) Evals

(07:55) Chain of Thought

(08:57) Coding

(11:08) Human Preference Evaluation

(11:37) What Is It?

(20:24) Doing Math Without Terrance Tao

(25:02) Doing Real Math with Terence Tao

(30:04) Positive Examples

(38:51) Skeptical Reactions

(42:32) Report from Janus World

(45:30) Same Old Silly Examples

(53:47) Latency

(55:14) Paths Forward Unrelated to Safety

(59:17) Safety Last

(01:07:06) Deception

(01:10:50) External Red Teaming

(01:11:23) Apollo's Red Teaming Finds Deceptive Alignment

(01:22:17) Preparedness Testing Finds Reward Hacking

(01:26:43) METR's Red Teaming

(01:29:52) What Are the Safety and Policy Implications?

The original text contained 37 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Following up on Alpha Fold, DeepMind has moved on to Alpha Proteo. We also got a rather simple prompt that can create a remarkably not-bad superforecaster for at least some classes of medium term events.

We did not get a new best open model, because that turned out to be a scam. And we don’t have Apple Intelligence, because it isn’t ready for prime time. We also got only one very brief mention of AI in the debate I felt compelled to watch.

What about all the apps out there, that we haven’t even tried? It's always weird to get lists of ‘top 50 AI websites and apps’ and notice you haven’t even heard of most of them.

Table of Contents

---

Outline:

(00:44) Language Models Offer Mundane Utility

(03:40) Language Models Don’t Offer Mundane Utility

(05:43) Predictions are Hard Especially About the Future

(12:57) Early Apple Intelligence

(15:27) On Reflection It's a Scam

(21:34) Deepfaketown and Botpocalypse Soon

(23:08) They Took Our Jobs

(28:42) The Time 100 People in AI

(32:11) The Art of the Jailbreak

(32:47) Get Involved

(33:12) Alpha Proteo

(43:14) Introducing

(44:23) In Other AI News

(46:41) Quiet Speculations

(50:40) The Quest for Sane Regulations

(53:12) The Week in Audio

(54:28) Rhetorical Innovation

(55:48) Aligning a Smarter Than Human Intelligence is Difficult

(56:05) People Are Worried About AI Killing Everyone

(58:38) Other People Are Not As Worried About AI Killing Everyone

(59:56) Six Boats and a Helicopter

(01:06:47) The Lighter Side

The original text contained 8 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.