There has been a renewal of discussion on how much hope we should have of an unaligned AGI leaving humanity alive on Earth after a takeover. When this topic is discussed, the idea of using simulation arguments or acausal trade to make the AI spare our lives often come up. These ideas have a long history. The first mention I know of comes from Rolf Nelson in 2007 on an SL4 message board, the idea later makes a brief appearance in Superintelligence under the name of Anthropic Capture, and came up on LessWrong last time as recently as a few days ago. In response to these, Nate Soares wrote Decision theory does not imply that we get to have nice things, arguing that decision theory is not going to save us, and that we can't bamboozle a superintelligence into submission by clever simulation arguments.

However, none of the [...]

---

Outline:

(01:15) A proposal for humanity in the Future

(03:50) What does the AI do?

(05:42) Is this the same as acausal trade?

(07:32) Response to Nates arguments in his post

(12:05) Nates arguments in the comments

(12:54) 1. We might just have a very low chance of solving alignment, so the AI doesnt need to take seriously the possibility of humans simulating it.

(14:12) 2. The successful human civilization would need to guess correctly what random thing an AI developing in a different Universe branch might value, and this is possibly infeasible.

(15:17) 3. Maybe the successful human civilization could pay for our salvation, but they will choose to spend their resources on other things.

(18:25) What should we bargain for?

(21:49) Can we get more than this?

(24:49) Other possible types of AI values

(25:23) Ethical considerations

(26:54) Ways this hope could fail

(30:24) 2. The true cost might be not just a planet, but the delay.

(32:44) 3. It might be very hard to create simulations that are indistinguishable from reality.

(33:18) 4. There are just too many possible simulators out there with too many different goals.

(37:51) 6. The AI just doesnt take the simulation hypothesis seriously.

(38:35) 7. A lot of people might be killed during takeover.

(39:44) Are we in a simulation? What should we do?

(43:34) Conclusion

The original text contained 34 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.