Epistemic status: very rough! Spent a couple of days reading the Gradual Disempowerment paper and thinking about my view on it. Won’t spend longer on this, so am sharing rough notes as is

Summary

• I won’t summarise the paper here! If you’re not familiar with it, I recommend reading the paper's exec summ before continuing. 
• There are lots of things I like about the paper! For example:
  • It's a nice lens on the role that structural and competitive dynamics play in threat models from misalignment and power concentration. 
  • It makes a great point: cultural evolution has historically been constrained to parts of “culture space” that keep humans economically productive (and thus alive!). Once AI obsolete human labour, that constraint won’t apply. So unconstrained cultural evolution becomes more scary.
  • It's a nice reminder that there are possible paths to human disempowerment that happen without AI being explicitly power-seeking. Though I think these [...]

---

Outline:

(00:23) Summary

(10:55) Things I agree with

(13:09) Points of disagreement

(20:35) Steelman of GD

(29:45) Steelman of GD even with the alignment problem solved?

(34:07) Especially rough thoughts

---

First published:

Source:

---

Narrated by TYPE III AUDIO.