https://astralcodexten.substack.com/p/willpower-human-and-machine

Two paragraphs from the mesa-optimizers post, which I quoted again in the adaptation-executors post:

Consider evolution, optimizing the fitness of animals. For a long time, it did so very mechanically, inserting behaviors like "use this cell to detect light, then grow toward the light" or "if something has a red dot on its back, it might be a female of your species, you should mate with it". As animals became more complicated, they started to do some of the work themselves. Evolution gave them drives, like hunger and lust, and the animals figured out ways to achieve those drives in their current situation. Evolution didn't mechanically instill the behavior of opening my fridge and eating a Swiss Cheese slice. It instilled the hunger drive, and I figured out that the best way to satisfy it was to open my fridge and eat cheese.

And:

Mesa-optimizers would have an objective which is closely correlated with their base optimizer, but it might not be perfectly correlated. The classic example, again, is evolution. Evolution "wants" us to reproduce and pass on our genes. But my sex drive is just that: a sex drive. In the ancestral environment, where there was no porn or contraceptives, sex was a reliable proxy for reproduction; there was no reason for evolution to make me mesa-optimize for anything other than "have sex". Now in the modern world, evolution's proxy seems myopic - sex is a poor proxy for reproduction. I know this and I am pretty smart and that doesn't matter. That is, just because I'm smart enough to know that evolution gave me a sex drive so I would reproduce - and not so I would have protected sex with somebody on the Pill - doesn't mean I immediately change to wanting to reproduce instead. Evolution got one chance to set my value function when it created