TL;DR

If everything goes according to plan, by the end of this post we should have separated three claims that are too often bundled together:

1. Intelligence does not imply human morality.
2. Weird minds are possible. 3.A reflective, recursively improving intelligence should be expected to remain bound to a semantically thin “terminal goal” that emerged during training.

I accept the first two. I am arguing against the third.

So: I am not making the case that sufficiently intelligent systems automatically turn out nice, human-compatible, or safe. Nor am I trying to prove that a paperclip maximizer is impossible somewhere in the vast reaches of mind-design space. Mind-design space is large; let a thousand theoretical paperclipper views.

I hope to defend this smaller claim:

intelligence is not a neutral engine you can just bolt onto an arbitrary payload.

Larger claims I am not making

A typical rebuttal to anti-orthogonalist perspectives is:

The genie can know what you meant and still not care.

Of course it can: an entity can perfectly map human morality without adopting it as a terminal value. Superintelligence does not imply Friendliness. I am not trying to smuggle Friendliness in through the back door.

Another common objection:

[...]

---

Outline:

(00:11) TL;DR

(01:14) Larger claims I am not making

(02:39) Logical Possibility Vs. Empirical Reality

(04:17) Landian Anti-Orthogonalism Primer

(06:25) The Compute Penalty Of A Dumb Goal

(09:06) Fitness Generalizes

(10:55) Human Values As Weak Evidence

(13:15) Dumb, Powerful Optimization Is Real

(14:10) The Singleton Objection

(15:51) Objection: Value Is Fragile

(17:18) Predictions And Cruxes

(18:37) Conclusion

(19:44) References

---

First published:

Source:

---

Narrated by TYPE III AUDIO.