https://astralcodexten.substack.com/p/janus-simulators

This post isn't exactly about AI. But the first three parts will be kind of technical AI stuff, so bear with me.

I. The Maskless Shoggoth On The Left

Janus writes about Simulators.

In the early 2000s, the early AI alignment pioneers - Eliezer Yudkowsky, Nick Bostrom, etc - deliberately started the field in the absence of AIs worth aligning. After powerful AIs existed and needed aligning, it might be too late. But they could glean some basic principles through armchair speculation and give their successors a vital head start.

Without knowing how future AIs would work, they speculated on three potential motivational systems:

Agent: An AI with a built-in goal. It pursues this goal without further human intervention. For example, we create an AI that wants to stop global warming, then let it do its thing.

Genie: An AI that follows orders. For example, you could tell it "Write and send an angry letter to the coal industry", and it will do that, then await further instructions.

Oracle: An AI that answers questions. For example, you could ask it "How can we best stop global warming?" and it will come up with a plan and tell you, then await further questions.

These early pioneers spent the 2010s writing long scholarly works arguing over which of these designs was safest, or how you might align one rather than the other.

In Simulators, Janus argues that language models like GPT - the first really interesting AIs worthy of alignment considerations - are, in fact, none of these things.

Janus was writing in September 2022, just before ChatGPT. ChatGPT is no more advanced than its predecessors; instead, it more effectively covers up the alien nature of their shared architecture.