Epistemic status: We really need to know. (And I have an opinionated answer.)

TL;DR: Comparing person-years of effort, I argue that AI Safety seems harder than for steam engines, but probably less hard than the Apollo program or . I discuss why I suspect superalignment might not be super-hard. My has come down over the last half-decade, primarily because of properties of LLMs, and progress we’ve made in aligning them: I explain why certain previous concerns don’t apply to LLMs, and summarize what I see as key developments in Alignment. I guesstimate we might be about 10%–20% done. Given the rate of progress, on my ASI timelines, it still doesn’t look we’re on track to be done in time, and I’m not comfortable about having AGI just align itself unsupervised, so I propose we aim to more-than-double the field's growth rate, and if possible also buy ourselves some more time.

There's a well-known diagram from a tweet by Chris Olah of Anthropic:

It would be marvelous to know what the actual difficulty is, out of those five labeled difficulty categories (ideally, exactly where it lies on that spectrum). This is a major crux that explains a large part of [...]

---

Outline:

(03:05) My Opinion

(04:16) 1. Calibrating the Scale

(09:10) 1.1. How Much Alignment Work Have we Done So Far?

(12:43) Trivial

(13:07) Steam Engine

(15:20) Apollo

(18:05) P vs. NP

(22:41) 1.2. Why I Think Superalignment Might Not be Completely Different

(29:15) 2. My Answer

(31:09) 3. My Reasons for Optimism

(33:05) 3.1. LLMs are Actually a Pretty Good Kind of AI to Align

(40:11) Human Values are Complex and Fragile ✘

(44:59) Ontology Mismatch / The Diamond Maximization Problem ✘

(49:07) Bad Reward Functions / Reward Hacking / Insufficient Supervision ~

(50:36) LLMs Are Just Way Too Easy to Control ~

(59:59) 3.2. Concerns That Didnt Hold Up

(01:00:04) Coping with Goodharts Law is Inherent to the Scientific Method

(01:05:29) 3.3. Pieces of Luck

(01:07:32) The Unreasonable Effectiveness of Chain-of-Thought

(01:17:50) Activation Oracles

(01:19:56) True Confessions

(01:23:36) 3.4. Actual Progress in Alignment

(01:24:47) Alignment Pretraining

(01:26:57) Personas

(01:29:33) Interpretability

(01:34:08) Model Organisms

(01:42:35) AI Control

(01:44:03) Weak-to-Strong Generalization and Scalable Oversight

(01:45:27) RLHF and Constitutional AI

(01:46:42) 3.5. Net Effect on my P(DOOM)

(01:48:00) 4. So, Are We Doomed?

The original text contained 25 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.