TL;DR: using a simple mixed strategy, LDT can give in to threats, ultimatums, and commitments - while incentivizing cooperation and fair[1] splits instead.

This strategy made it much more intuitive to many people I've talked to that smart agents probably won't do weird everyone's-utility-eating things like threatening each other or participating in commitment races.

1. The Ultimatum game

This part is taken from planecrash[2][3].

You're in the Ultimatum game. You're offered 0-10 dollars. You can accept or reject the offer. If you accept, you get what's offered, and the offerer gets $(10-offer). If you reject, both you and the offerer get nothing.

The simplest strategy that incentivizes fair splits is to accept everything ≥ 5 and reject everything < 5. The offerer can't do better than by offering you 5. If you accepted offers of 1, the offerer that knows this would always offer you 1 and get 9 [...]

---

Outline:

(00:28) 1. The Ultimatum game

(06:17) 2. Threats, commitments, and ultimatums

(06:41) 2.1 Game of Chicken

(07:41) 2.2 Stones

(08:00) 2.3 What if I dont know the other players payoffs?

(08:25) 2.4 What if the other player makes a commitment before I make any decisions?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

[Conflict of interest disclaimer: We are FutureSearch, a company working on AI-powered forecasting and other types of quantitative reasoning. If thin LLM wrappers could achieve superhuman forecasting performance, this would obsolete a lot of our work.]

Widespread, misleading claims about AI forecasting

Recently we have seen a number of papers – (Schoenegger et al., 2024, Halawi et al., 2024, Phan et al., 2024, Hsieh et al., 2024) – with claims that boil down to “we built an LLM-powered forecaster that rivals human forecasters or even shows superhuman performance”.

These papers do not communicate their results carefully enough, shaping public perception in inaccurate and misleading ways. Some examples of public discourse:

• Ethan Mollick (>200k followers) tweeted the following about the paper Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy by Schoenegger et al.:

• A post on Marginal Revolution with the title and abstract [...]

---

Outline:

(00:24) Widespread, misleading claims about AI forecasting

(03:02) What does human-level or superhuman forecasting mean?

(04:08) Red flags for claims to (super)human AI forecasting accuracy

(06:42) Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy (Schoenegger et al., 2024)

(09:14) Approaching Human-Level Forecasting with Language Models (Halawi et al., 2024)

(11:10) Reasoning and Tools for Human-Level Forecasting (Hsieh et al., 2024)

(12:50) LLMs Are Superhuman Forecasters (Phan et al., 2024)

(15:19) Takeaways

(16:17) So how good are AI forecasters?

The original text contained 1 footnote which was omitted from this narration.

The original text contained 6 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Following up on Alpha Fold, DeepMind has moved on to Alpha Proteo. We also got a rather simple prompt that can create a remarkably not-bad superforecaster for at least some classes of medium term events.

We did not get a new best open model, because that turned out to be a scam. And we don’t have Apple Intelligence, because it isn’t ready for prime time. We also got only one very brief mention of AI in the debate I felt compelled to watch.

What about all the apps out there, that we haven’t even tried? It's always weird to get lists of ‘top 50 AI websites and apps’ and notice you haven’t even heard of most of them.

Table of Contents

---

Outline:

(00:44) Language Models Offer Mundane Utility

(03:40) Language Models Don’t Offer Mundane Utility

(05:43) Predictions are Hard Especially About the Future

(12:57) Early Apple Intelligence

(15:27) On Reflection It's a Scam

(21:34) Deepfaketown and Botpocalypse Soon

(23:08) They Took Our Jobs

(28:42) The Time 100 People in AI

(32:11) The Art of the Jailbreak

(32:47) Get Involved

(33:12) Alpha Proteo

(43:14) Introducing

(44:23) In Other AI News

(46:41) Quiet Speculations

(50:40) The Quest for Sane Regulations

(53:12) The Week in Audio

(54:28) Rhetorical Innovation

(55:48) Aligning a Smarter Than Human Intelligence is Difficult

(56:05) People Are Worried About AI Killing Everyone

(58:38) Other People Are Not As Worried About AI Killing Everyone

(59:56) Six Boats and a Helicopter

(01:06:47) The Lighter Side

The original text contained 8 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 76 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Refer to the arXiv preprint for full content. This post is a lighter, 15-minute version.

Abstract

• Existing activation steering methods alter LLM behavior indiscriminately, limiting their practical applicability in settings where selective responses are essential, such as content moderation or domain-specific assistants. 
• We propose Conditional Activation Steering (CAST), which analyzes LLM activation patterns during inference to selectively apply or withhold activation steering based on the input context. 
• Using CAST, one can systematically control LLM behavior with rules like “if input is about hate speech or adult content, then refuse” or “if input is not about legal advice, then refuse.” 
• This allows for selective modification of responses to specific content while maintaining normal responses to other content, all [...]

---

Outline:

(00:25) Abstract

(01:26) Introduction

(04:35) Background

(06:55) Conditional Activation Steering

(12:57) Conditioning Refusal: Selectively Steering on Harmful Prompts

(15:40) Programming Refusal: Logical Composition of Refusal Condition

(22:52) Discussion

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I first learned about cryonics when I read Eliezer and Robin's posts about it on Overcoming Bias years ago. I got cryopilled. Somewhat amazingly to me, I'm now a researcher in this field. So I thought this community might be interested to know that I was one of several co-authors on a paper just published in Frontiers in Medical Technology, titled "Structural brain preservation: a potential bridge to future medical technologies".

In this paper, we propose reframing cryonics as a type of structural brain preservation, focusing on maintaining the brain's physical structure that encodes memories and personality, rather than making the focus about low-temperature storage per se.

We explore what brain structures likely need to be preserved to retain long-term memories and other valued aspects of personal identity. We then review different methods of brain preservation, including cryopreservation, aldehyde-stabilized cryopreservation, fluid preservation, and fixation followed by polymer embedding. The [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

People often attack frontier AI labs for "hypocrisy" when the labs admit publicly that AI is an extinction threat to humanity. Often these attacks ignore the difference between various kinds of hypocrisy, some of which are good, including what I'll call "reformative hypocrisy". Attacking good kinds of hypocrisy can be actively harmful for humanity's ability to survive, and as far as I can tell we (humans) usually shouldn't do that when our survival is on the line. Arguably, reformative hypocrisy shouldn't even be called hypocrisy, due to the negative connotations of "hypocrisy". That said, bad forms of hypocrisy can disguise itself as the reformative kind for long periods, so it's important to pay enough attention to hipcrosity to actually figure out what kind it is.

Here's what I mean, by way of examples:

***

0. No Hypocrisy —
Lab: "Building AGI without regulation shouldn't be allowed. Since there's no [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Formalizing the Informal

One way to view MIRI's Agent Foundations research is that it saw the biggest problem in AI safety as "human preferences are informal, but we need to somehow get formal guarantees about them" -- and so, in response, it set out to make a formal-informal bridge.

Recently, I’ve been thinking about how we might formally represent the difference between formal and informal. My prompt is something like: if we assume that classical probability theory applies to “fully formal” propositions, how can we generalize it to handle “informal” stuff?

I’m going to lead a discussion on this tomorrow, Wednesday Sept. 11, at 11am EDT (7am Pacific, 4pm UK).

Discord Event link (might not work for most people):

https://discord.com/events/1237103274591649933/1282859362125352960

Zoom link (should work for everyone):

https://us06web.zoom.us/j/6274543940?pwd=TGZpY3NSTUVYNHZySUdCQUQ5ZmxQQT09

You can support my work on Patreon.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I am posting this now largely because it is the right place to get in discussion of unrealized capital gains taxes and other campaign proposals, but also there is always plenty of other stuff going on. As always, remember that there are plenty of really stupid proposals always coming from all sides. I’m not spending as much time talking about why it's awful to for example impose gigantic tariffs on everything, because if you are reading this I presume you already know.

The Biggest Economics Problem

The problem, perhaps, in a nutshell:

Tess: like 10% of people understand how markets work and about 10% deeply desire and believe in a future that's drastically better than the present but you need both of these to do anything useful and they’re extremely anticorrelated so we’re probably all fucked.

In my world the two are correlated. If you [...]

---

Outline:

(00:31) The Biggest Economics Problem

(02:30) No Good Very Bad Capital Gains Tax Proposals

(14:13) Hot Tip

(17:11) Gouging at the Grocer

(20:08) Noncompetes Nonenforcement Cannot Compete With Courts

(20:42) We Used to Be Poor

(25:13) Everywhere But in the Productivity Statistics

(26:32) They Don’t Make ‘Em Like They Used To

(30:29) Disclosure of Wages Causes Lower Wages

(32:30) In Other Economic News

(36:50) The Efficient Market Hypothesis is (Even More) False

The original text contained 4 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

(This was supposed to be on Thursday but I forgot to cross-post)

Will AI ever make art? Fully do your coding? Take all the jobs? Kill all the humans?

Most of the time, the question comes down to a general disagreement about AI capabilities. How high on a ‘technological richter scale’ will AI go? If you feel the AGI and think capabilities will greatly improve, then AI will also be able to do any particular other thing, and arguments that it cannot are almost always extremely poor. However, if frontier AI capabilities level off soon, then it is an open question how far we can get that to go in practice.

A lot of frustration comes from people implicitly making the claim that general AI capabilities will level off soon, usually without noticing they are doing that. At its most extreme, this is treating AI as [...]

---

Outline:

(01:53) Language Models Offer Mundane Utility

(03:05) Language Models Don’t Offer Mundane Utility

(04:03) Fun with Image Generation

(06:14) Copyright Confrontation

(07:09) Deepfaketown and Botpocalypse Soon

(12:09) They Took Our Jobs

(13:46) Time of the Season

(16:53) Get Involved

(17:15) Introducing

(19:08) In Other AI News

(19:51) Quiet Speculations

(30:10) A Matter of Antitrust

(37:34) The Quest for Sane Regulations

(40:06) The Week in Audio

(47:40) Rhetorical Innovation

(53:09) The Cosmos Institute

(56:21) The Alignment Checklist

(01:00:34) People Are Worried About AI Killing Everyone

(01:02:39) Other People Are Not As Worried About AI Killing Everyone

(01:06:21) Five Boats and a Helicopter

(01:09:07) Pick Up the Phone

(01:12:58) The Lighter Side

The original text contained 9 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.