Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hands-On Experience Is Not Magic, published by Thane Ruthenis on May 27, 2023 on The AI Alignment Forum.

Here are some views, oftentimes held in a cluster:

You can't make strong predictions about what superintelligent AGIs will be like. We've never seen anything like this before. We can't know that they'll FOOM, that they'll have alien values, that they'll kill everyone. You can speculate, but making strong predictions about them? That can't be invalid.

You can't figure out how to align an AGI without having an AGI on-hand. Iterative design is the only approach to design that works in practice. Aligning AGI right on the first try isn't simply hard, it's impossible, so racing to build an AGI to experiment with is the correct approach for aligning it.

An AGI cannot invent nanotechnology/brain-hacking/robotics/[insert speculative technology] just from the data already available to humanity, then use its newfound understanding to build nanofactories/take over the world/whatever on the first try. It'll have to engage in extensive, iterative experimentation first, and there'll be many opportunities to notice what it's doing and stop it.

More broadly, you can't genuinely generalize out of distribution. The sharp left turn is a fantasy — you can't improve without the policy gradient, and unless there's someone holding your hand and teaching you, you can only figure it out by trial-and-error. Thus, there wouldn't be genuine sharp AGI discontinuities.

There's something special about training by SGD, and the "inscrutable" algorithms produced this way. They're a specific kind of "connectivist" algorithms made up of an inchoate mess of specialized heuristics. This is why interpretability is difficult — it involves translating these special algorithms into a more high-level form — and indeed, it's why AIs may be inherently uninterpretable!

You can probably see the common theme here. It holds that learning by practical experience (henceforth LPE) is the only process by which a certain kind of cognitive algorithms can be generated. LPE is the only way to become proficient in some domains, and the current AI paradigm works because it implements this kind of learning, and it only works inasmuch as it implements this kind of learning.

All in all, it's not totally impossible. I myself had suggested that some capabilities may only be implementable via one algorithm and one algorithm only.

But I think this is false, in this case. And perhaps, when put this way, it already looks false to you as well.

If not, let's dig into the why.

A Toy Formal Model

What is a "heuristic", fundamentally speaking? It's a recorded statistical correlation — the knowledge that if you're operating in some environment E with the intent to achieve some goal G, taking the action A is likely to lead to achieving that goal.

As a toy formality, we can say that it's a structure of the following form:

The question is: what information is necessary for computing h? Clearly you need to know E and G — the structure of the environment and what you're trying to do there. But is there anything else?

The LPE view says yes: you also need a set of "training scenarios" S={EA1, ..., EAn}, where the results of taking various actions Ai on the environment are shown. Not because you need to learn the environment's structure — we're already assuming it's known. No, you need them because... because...

Perhaps I'm failing the ITT here, but I think the argument just breaks down at this step, in a way that can't be patched. It seems clear, to me, that E itself is entirely sufficient to compute h, essentially by definition. If heuristics are statistical correlations, it should be sufficient to know the statistical model of the environment to generate one!

Toy-formally, P(h|ES)=P(h|E). Once the environment's structure is known, you gain no...