LessWrong posts by zvi

Once again we’ve reached the point where the weekly update needs to be split in two. Thus, the alignment and policy coverage will happen tomorrow. Today covers the rest.

The secret big announcement this week was Claude for Chrome. This is a huge deal. It will be rolling out slowly. When I have access or otherwise know more, so will you.

The obvious big announcement was Gemini Flash 2.5 Image. Everyone agrees this is now the clear best image editor available. It is solid as an image generator, but only as one among many on that front. Editing abilities, including its ability to use all its embedded world knowledge, seem super cool.

The third big story was the suicide of Adam Raine, which appears to have been enabled in great detail by ChatGPT. His parents are suing OpenAI and the initial facts very much do not [...]

---

Outline:

(01:14) Language Models Offer Mundane Utility

(05:42) Language Models Don't Offer Mundane Utility

(06:26) Huh, Upgrades

(07:11) Fun With Image Generation

(12:53) On Your Marks

(16:13) Water Water Everywhere

(20:13) Get My Agent On The Line

(25:21) Choose Your Fighter

(27:52) Deepfaketown and Botpocalypse Soon

(31:15) You Drive Me Crazy

(36:51) The Worst Tragedy So Far

(43:22) Unprompted Attention

(44:27) Copyright Confrontation

(45:00) The Art of the Jailbreak

(45:13) Get Involved

(45:34) Introducing

(46:23) In Other AI News

(49:10) Show Me the Money

(51:27) Quiet Speculations

(51:51) Rhetorical Innovation

(01:01:46) The Week in Audio

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In the wake of the confusions around GPT-5, this week had yet another round of claims that AI wasn’t progressing, or AI isn’t or won’t create much value, and so on. There were reports that one study in particular impacted Wall Street, and as you would expect it was not a great study. Situational awareness is not what you’d hope.

I’ve gathered related coverage here, to get it out of the way before whatever Google is teasing (Gemini 3.0? Something else?) arrives to potentially hijack our attention.

We’ll start with the MIT study on State of AI in Business, discuss the recent set of ‘AI is slowing down’ claims as part of the larger pattern, and then I will share a very good attempted explanation from Steven Byrnes of some of the ways economists get trapped into failing to look at what future highly capable AIs would [...]

---

Outline:

(01:02) Standing On Lack of AI Business

(04:23) Claims Of Zero Returns Do Not Mean What You Might Think

(05:35) Crossing The Divide

(09:23) Unrealistic (or Premature) Expectations

(10:40) Claims Of Prohibitive Lock-In Effects Are Mostly Hype

(12:10) Nothing Ever Changes About Claims That Nothing Ever Changes

(14:20) Ask What AI Can Do For You

(15:48) The Pattern Of Claiming a Slowdown Continues

(18:28) A Sensible Move

(18:39) Who Deserves The Credit And Who Deserves The Blame

(20:39) Mistakes From Applying Standard Economics To Future AIs

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I happily admit I am deeply confused about consciousness.

I don’t feel confident I understand what it is, what causes it, which entities have it, what future entities might have it, to what extent it matters and why, or what we should do about these questions. This applies both in terms of finding the answers and what to do once we find them, including the implications for how worried we should be about building minds smarter and more capable than human minds.

Some people respond to this uncertainty by trying to investigate these questions further. Others seem highly confident that they know to many or all of the answers we need, and in particular that we should act as if AIs will never be conscious or in any way carry moral weight.

Claims about all aspects of the future of AI are often highly motivated.

[...]

---

Outline:

(02:40) Asking The Wrong Questions

(08:47) How To Usefully Predict And Interact With an AI

(11:23) The Only Thing We Have To Fear

(14:45) Focused Fixation

(20:13) What Even Is Consciousness Anyway?

(23:03) Most People Who Think An AI Is Currently Conscious Are Thinking This For Unjustified Reasons

(24:42) Mustafa's Case Against AI Consciousness

(36:20) Where Are They Going Without Ever Knowing The Way

(37:38) When We Talk About AI Consciousness Things Get Weird

(41:03) We Don't Talk About AI Consciousness

(44:41) Some Things To Notice

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Why We Haven’t Seen v4 or r2

---

Outline:

(00:24) Why We Haven't Seen v4 or r2

(01:49) Introducing DeepSeek v3.1

(04:34) Signs of Life

(06:57) How Should We Update?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

One potentially big event was that DeepSeek came out with v3.1. Initial response was very quiet, but this is DeepSeek and there are some strong scores especially on SWE and people may need time to process the release. So I’m postponing my coverage of this to give us time to learn more.

Meta is restructuring its AI operations, including a hiring freeze. Some see this as some sign of an AI pullback. I don’t think that is right.

Nor do I think what they are doing with their Ai companions is right, as we got a look inside their 200 page document of what they think is acceptable. I wrote about current AI Companion Conditions at Meta and also xAI.

The weirdest event of the week was America and China both self-sabotaging on chips. America is trying to sell Nvidia H20s to China and looks open [...]

---

Outline:

(01:44) Language Models Offer Mundane Utility

(06:58) Language Models Don't Offer Mundane Utility

(12:54) Huh, Upgrades

(17:41) Absurd Sycophancy

(21:19) The Real Alignment Problem Is We Don't Know How To Align Models

(29:15) Unprompted Suggestions

(29:55) On Your Marks

(34:11) Choose Your Fighter

(36:46) Preserve Our History

(40:48) Autonomous Friendly Robots

(42:00) Deepfaketown and Botpocalypse Soon

(46:25) Oops I Did It Again

(49:38) You Drive Me Crazy

(56:12) They Took Our Jobs

(59:28) Get Involved

(59:42) Introducing

(01:00:02) In Other AI News

(01:04:19) Show Me the Money

(01:08:59) Lol We're Meta

(01:13:33) Quiet Speculations

(01:16:14) The Quest for Sane Regulations

(01:16:35) Chip City

(01:27:33) The Week in Audio

(01:35:15) Rhetorical Innovation

(01:42:09) Misaligned!

(01:42:44) Open Models

(01:45:48) AI Model Welfare

(01:48:41) Aligning a Smarter Than Human Intelligence is Difficult

(01:51:08) People Are Worried About AI Killing Everyone

(01:55:02) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The conditions are: Lol, we’re Meta. Or lol we’re xAI.

This expands upon many previous discussions, including the AI Companion Piece.

Lol We’re Meta

I said that ‘Lol we’re Meta’ was their alignment plan.

It turns out their alignment plan was substantially better or worse (depending on your point of view) than that, in that they also wrote down 200 pages of details of exactly how much lol there would be over at Meta. Every part of this was a decision.

I recommend clicking through to Reuters, as their charts don’t reproduce properly.

Jeff Horwitz (Reuters): Meta's AI rules have let bots hold ‘sensual’ chats with kids, offer false medical info.

An internal Meta Platforms document detailing policies on chatbot behavior has permitted the company's artificial intelligence creations to “engage a child in conversations that are romantic or sensual,” generate false medical [...]

---

Outline:

(00:21) Lol We're Meta

(04:15) If 'Lol We're Meta' It Is Good To Write That Down

(06:06) We Can't Help But Notice What Meta Wrote Down

(11:55) xAI and Meta Companion Offerings Are Differently Awful

(16:40) You're The Only One That Groks Me

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I got suckered into paying attention to multiple non-AI political stories this month: The shooting of the messenger, in violation of the most sacred principles, via firing the head of the USA's Bureau of Labor Statistics, and the Online Safety Bill in the UK.

As a reminder, feel no obligation whatsoever to engage with either of these.

There are tons of other things worth paying attention to that are not that.

Highly Effective Altruism

I realize philanthropy has been handed quite a few sinking ships lately, but if you exclude AI there is one crisis I would prioritize above the rest, which is mRNA.

Which is: We have entered the War on Cancer, future pandemics and other diseases on the side of cancer and future pandemics and those other diseases, because we decided to hand this power over to RFK Jr.

As Rick [...]

---

Outline:

(00:36) Highly Effective Altruism

(07:06) Variously Effective Altruism

(10:39) Bad News

(15:51) Whisper Networks

(17:36) Opportunity Knocks

(19:06) Government Working

(20:45) Censorship In The UK

(45:50) Don't Shoot the Messenger

(59:57) Jones Act Watch

(01:00:12) The Incorrect Contrarian Cluster

(01:01:53) Talent Search

(01:04:19) While I Cannot Condone This

(01:11:41) Predictions Are Hard Especially About The Future

(01:14:01) Good News, Everyone

(01:15:49) For Your Entertainment

(01:19:42) Game Theory

(01:20:39) Gamers Gonna Game Game Game Game Game

(01:28:48) I Was Promised Flying Self-Driving Cars

(01:34:00) Sports Go Sports

(01:36:17) Antisocial Media

(01:41:50) Stop Deboosting Links On Twitter

(01:49:10) Technology Advances

(01:52:35) The Lighter Side

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Everyone agrees that the release of GPT-5 was botched. Everyone can also agree that the direct jump from GPT-4o and o3 to GPT-5 was not of similar size to the jump from GPT-3 to GPT-4, that it was not the direct quantum leap we were hoping for, and that the release was overhyped quite a bit.

GPT-5 still represented the release of at least three distinct models: GPT-5-Fast, GPT-5-Thinking and GPT-5-Pro, at least two and likely all three of which are SoTA (state of the art) within their class, along with GPT-5-Auto.

The problem is that the release was so botched that OpenAI is now experiencing a Reverse DeepSeek Moment – all the forces that caused us to overreact to DeepSeek's r1 are now working against OpenAI in reverse.

This threatens to give Washington DC and its key decision makers a very false impression of [...]

---

Outline:

(01:31) GPT-5: The Reverse DeepSeek Moment

(06:18) Did You Know AI Is Making Rapid Progress?

(08:42) No We Are Not Yet Hitting A Wall

(12:34) Models Making Money And Being Useful Does Not Mean Less Progress

(20:19) We Could Massively Screw This All Up

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In honor of Nate Silver's analysis of when to leave for the airport, and because it's been an intense week, I thought I’d offer my thoughts on various related questions.

Buying The Ticket

As far as I can tell, the major booking portals for tickets are all basically the same. I’ve been using Orbitz for a long time because I’m used to the interface, it is clean and I have confidence it works. The times I checked Kayak and so on they all seemed to be exactly the same.

I still book tickets manually rather than using an AI agent. There isn’t much time to plausibly save and by the time I fully express preferences and enter my information anew I might as well have just done it myself. It also means I look at alternatives, which helps me keep tabs.

My heuristic is to [...]

---

Outline:

(00:29) Buying The Ticket

(02:09) Planning To Spend Time At The Airport

(03:41) When To Head To The Airport

(08:35) Packing Your Own Bags and Boarding the Plane

(09:20) All Aboard

(09:35) What Happens After The Airport

(10:23) What To Do In The Air

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.