The epistemic status thing

Please read this section because it is not a disclaimer for the sake of it.

I think the model I describe in this post is more likely wrong than right. I also think the probability of it being substantially correct is non-negligible and even more than that, and that this alone warrants writing it up and thinking about it seriously.

Throughout this post, I will write "X is the case" rather than "it seems plausible that X might be the case, though I am not certain." Every claim should be mentally prefixed with the appropriate hedge. I'm dropping the hedges not because I'm confident, but because a text where every sentence contains "it seems plausible that" becomes unreadable without actually making anyone more calibrated.

I should also flag that I'm writing this from a state of considerable frustration with the current state of affairs, and I this frustration may be load-bearing in places where it shouldn't be.

This is not about AI

AI risk is, at most, one illustration of the phenomenon I'm describing here, and not a particularly privileged one. Everything in this post would hold in a world where artificial intelligence [...]

---

Outline:

(00:10) The epistemic status thing

(01:03) This is not about AI

(01:51) The central claim: extinctive pressure

(04:54) Optimization does not target survival

(05:52) The worst of the best possible worlds

(09:10) Local environments are not stable, and we are actively destabilizing ours

(09:38) The entropy frame

(10:47) Feedback loops for survival are not tight

(12:02) Survival conflicts with optimization

(15:06) The ability to foresee landscape changes does not reliably help

(16:04) Requirements for survival under optimization

(17:21) The civilization path is not ergodic

(18:57) Survival does not scale

(20:15) The process is fat-tailed

(21:45) In the absence of sufficiently strong corrective mechanisms, the multiplicative fat-tailed process drives the system to zero

(26:00) Instrumental convergence: the strongest counterargument, and why it probably fails in practice

(29:08) Every emerging civilization is around the dumbest possible civilization

(31:01) Intelligence is not the bottleneck

(32:39) Caring about survival is not the same as surviving

(34:01) The progress dilemma: there are no good options

(36:22) Indirect supporting arguments

(39:03) Summary: you are punished for caring about survival

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.