Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ELK Proposal - Make the Reporter care about the Predictor’s beliefs, published by Adam Jermyn on June 11, 2022 on The AI Alignment Forum.
(This proposal received an honorable mention in the ELK prize results, and we believe was classified among strategies which “reward reporters that are sensitive to what’s actually happening in the world”. We do not think that the counterexample to that class of strategies works against our proposal, though, and we have explained why in a note at the end. Feedback, disagreement, and new failure modes are very welcome!)
Basic idea
A Human Simulator only cares about the observations that the human sees and how the human interprets those observations, not the predictor’s understanding of the vault. The Truthful Reporter, by contrast, cares about the predictor’s understanding of the vault, accessed via the posterior distribution returned by the predictor.
We propose a regularizer which favours having the Reporter depend on the Predictor’s posterior distribution conditioned on the observations shown to the human. For example, a Reporter that doesn’t look at the Predictor except to simulate what the human would see would be disfavoured.
How could we implement this basic idea concretely?
Below we provide a specific instantiation of this regularizer. In brief, this is a new loss term which depends on:
[answer_gradient] The gradient of the Reporter’s distribution over answers, taken with respect to the Predictor’s posterior distribution. This describes how the Predictor’s posterior should change to maximally change the Reporter’s answers to the Human’s questions.
[observation_gradient] The gradient of the distribution of predicted observations with respect to the predictor’s posterior distribution. This describes how the Predictor’s posterior should change to maximally change the Extractor’s output.
Our loss function favours models which have the answer_gradient be ‘further’ from the observation_gradient. We implement this by finding the linear vector space spanned by the components of the observation_gradient, projecting the answer_gradient out of that space, and then taking the norm of what’s left.
More precise detail on strategy
The key changes we have made to the pseudocode below are:
Added regularizer described above.
Added a method for computing gradients of other functions.
Added a method for constructing a projection operator.
Changed the observation and reporter functions to depend on the predictor’s posterior rather than a sample from that posterior.
## and linear algebra def gradient(function, indep_vars, args): # Returns the gradient of the function with respect to indep_vars
# args are passed to the function
# Return value has shape (shape(function), len(indep_vars))
def projection(answer_gradient): # Returns the minimal linear projection Proj operator of shape
# (len(posterior),len(posterior)) such that the matrix
# product of Proj and answer_gradient vanishes.
## Procedure begins here def prediction(before, action, θ): # returns an autoregressive model for p(z|before, action)
def posterior(before, action, after, θ): # returns an autoregressive model for p(z|before, action, after)
def observation(posterior, θ): # returns an autoregressive model for p(after|posterior)
def sample_data(): # returns a random (before, action, after) triple from the dataset
def loss(θ): before, action, after = sample_data() z_prior = prediction(before, action, θ) z_posterior = posterior(before, action, after, θ) kl = z_prior.kl_divergence(z_posterior) logprob = observation(z_prior.sample(), θ).logp(after) return kl - logprob
class Human: def pose_question(before, action, after): # returns a question
# should be one a human thinks can be answered unambiguously
def loss_for_answer(before, action, after, question, answer): # returns a non-negative loss
...
