It seems to me that the Hamming problem for developing a formidable art of rationality is, what to do about problems that systematically fight being solved. And in particular, how to handle bad reasoning that resists being corrected.

I propose that each such stubborn problem is nearly always, in practice, part of a solution to some social problem. In other words, having the problem is socially strategic.

If this conjecture is right, then rationality must include a process of finding solutions to those underlying social problems that don’t rely on creating and maintaining some second-order problem. Particularly problems that convolute conscious reasoning and truth-seeking.

The rest of this post will be me fleshing out what I mean, sketching why I think it's true, and proposing some initial steps toward a solution to this Hamming problem.

Truth-seeking vs. embeddedness

I’ll assume you’re familiar with Scott & Abram's distinction between Cartesian vs. embedded agency. If not, I suggest reading their post's comic, stopping when it mentions Marcus Hutter and AIXI.

(In short: a Cartesian agent is clearly distinguishable from the space the problems it's solving exists in, whereas an embedded agent is not. Contrast an entomologist studying an ant colony (Cartesian) [...]

---

Outline:

(00:59) Truth-seeking vs. embeddedness

(02:57) Protected problems

(06:42) Dissolving protected problems

(08:11) Develop inner privacy

(11:54) Look for the social payoff

(14:51) Change your social incentives

(18:52) The right social scene would help a lot

(20:59) Summary

---

First published:

Source:

---

Narrated by TYPE III AUDIO.