A frame I am trying on:

When I say I'm worried about takeover by "AI superintelligence", I think the thing I mean by "intelligence" is "relentless, creative resourcefulness."

I think Eliezer argues something like "in the limit, superintelligence needs to include super-amounts-of Relentless, Creative Resourcefulness."

(because, if it didn't, it'd get stuck at some point, and then give up, instead of figuring out a way to deal with being stuck. And later, someone would build something more relentless, creative, and resourceful)

But, it's actually kind of interesting an important that you can accomplish some intellectual tasks without RCR. LLMs don't rely on it much at all (it seems to be the thing they are actively bad at). Instead, they work via "knowing a lot of stuff, and being good at pattern-matching their way into useful connections between stuff you want and stuff they know."

So it might be [...]

---

Outline:

(02:43) Examples

(02:46) Paul Graham on Startup Founders

(05:08) Richard Feynman

(06:47) Elon Musk

(08:10) Back to AI: Sable, in IABIED

(16:37) Notes on Sable

(18:00) Reflections from Rationality Training

(18:59) Buckling Up/Down

(20:01) Thinking Assistants

(21:16) Quiet Theaters

(23:14) One Shot Baba is You

(25:30) Takeaways

(26:13) Intelligence without RCR?

(27:23) What, if not agency?

(29:37) Abrupt Ending

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.