Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Supervise Process, not Outcomes, published by stuhlmueller on April 5, 2022 on LessWrong.
We can think about machine learning systems on a spectrum from process-based to outcome-based:
Process-based systems are built on human-understandable task decompositions, with direct supervision of reasoning steps.
Outcome-based systems are built on end-to-end optimization, with supervision of final results.
This post explains why Ought is devoted to process-based systems. The argument is:
In the short term, process-based ML systems have better differential capabilities: They help us apply ML to tasks where we don’t have access to outcomes. These tasks include long-range forecasting, policy decisions, and theoretical research.
In the long term, process-based ML systems help avoid catastrophic outcomes from systems gaming outcome measures and are thus more aligned.
Both process- and outcome-based evaluation are attractors to varying degrees: Once an architecture is entrenched, it’s hard to move away from it. This lock-in applies much more to outcome-based systems.
Whether the most powerful ML systems will primarily be process-based or outcome-based is up in the air.
So it’s crucial to push toward process-based training now.
There are almost no new ideas here. We’re reframing the well-known outer alignment difficulties for traditional deep learning architectures and contrasting them with compositional approaches. To the extent that there are new ideas, credit primarily goes to Paul Christiano and Jon Uesato.
We only describe our background worldview here. In a follow-up post, we’ll explain why we’re building Elicit, the AI research assistant.
The spectrum
Supervising outcomes
Supervision of outcomes is what most people think about when they think about machine learning. Local components are optimized based on an overall feedback signal:
SGD optimizes weights in a neural net to reduce its training loss
Neural architecture search optimizes architectures and hyperparameters to have low validation loss
Policy gradient optimizes policy neural nets to choose actions that lead to high expected rewards
In each case, the system is optimized based on how well it’s doing empirically.
MuZero is an example of a non-trivial outcome-based architecture. MuZero is a reinforcement learning algorithm that reaches expert-level performance at Go, Chess, and Shogi without human data, domain knowledge, or hard-coded rules. The architecture has three parts:
A representation network, mapping observations to states
A dynamics network, mapping state and action to future state, and
A prediction network, mapping state to value and distribution over next actions.
Superficially, this looks like an architecture with independently meaningful components, including a “world model” (dynamics network). However, because the networks are optimized end-to-end to jointly maximize expected rewards and to be internally consistent, they need not capture interpretable dynamics or state. It’s just a few functions that, if chained together, are useful for predicting reward-maximizing actions.
Neural nets are always in the outcomes-based regime to some extent: In each layer and at each node, they use the matrices that make the neural net as a whole work well.
Supervising process
If you’re not optimizing based on how well something works empirically (outcomes), then the main way you can judge it is by looking at whether it’s structurally the right thing to do (process).
For many tasks, we understand what pieces of work we need to do and how to combine them. We trust the result because of this reasoning, not because we’ve observed final results for very similar tasks:
Engineers and astronomers expect the James Webb Space Telescope to work because its deployment follows a well-understood plan, and it is built out of well-understo...
