Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Models Don't "Get Reward", published by Sam Ringer on December 30, 2022 on The AI Alignment Forum.
In terms of content, this has a lot of overlap with Reward is not the optimization target. I'm basically rewriting a part of that post in language I personally find clearer, emphasising what I think is the core insight.
When thinking about deception and RLHF training, a simplified threat model is something like this:
A model takes some actions.
If a human approves of these actions, the human gives the model some reward.
Humans can be deceived into giving reward in situations where they would otherwise not if they had more knowledge.
Models will take advantage of this so they can get more reward.
Models will therefore become deceptive.
Before continuing, I would encourage you to really engage with the above. Does it make sense to you? Is it making any hidden assumptions? Is it missing any steps? Can you rewrite it to be more mechanistically correct?
I believe that when people use the above threat model, they are either using it as shorthand for something else or they misunderstand how reinforcement learning works. Most alignment researchers will be in the former category. However, I was in the latter.
I was missing an important insight into how reinforcement learning setups are actually implemented. This lack of understanding led to lots of muddled thinking and general sloppiness on my part. I see others making the exact same mistake so I thought I would try and motivate a more careful use of language!
How Vanilla Reinforcement Learning Works
If I were to explain RL to my parents, I might say something like this:
You want to train your dog to sit.
You say "sit" and give your dog a biscuit if it sits.
Your dog likes biscuits, and over time it will learn it can get more biscuits by sitting when told to do so.
Biscuits have let you incentivise the behaviour you want.
We do the same thing with a computer by giving the computer "reward" when it does things we like. Over time, the computer will do more of the behaviour we like so it can get more reward.
Do you agree with this? Is this analogy flawed in any way?
I claim this is actually NOT how vanilla reinforcement learning works.The framing above views models as "wanting" reward, with reward being something models "receive" on taking certain actions. What actually happens is this:
The model takes a series of actions (which we collect across multiple "episodes").
After collecting these episodes, we determine how good the actions in each episode are using a reward function.
We use gradient descent to alter the parameters of the model so the good actions will be more likely and the bad actions will be less likely when we next collect some episodes.
The insight is that the model itself never "gets" the reward. Reward is something used separately from the model/environment.
To motivate this, let's view the above process not from the vantage point of the overall training loop but from the perspective of the model itself. For the purposes of demonstration, let's assume the model is a conscious and coherent entity. From it's perspective, the above process looks like:
Waking up with no memories in an environment.
Taking a bunch of actions.
Suddenly falling unconscious.
Waking up with no memories in an environment.
Taking a bunch of actions.
and so on.....
The model never "sees" the reward. Each time it wakes up in an environment, its cognition has been altered slightly such that it is more likely to take certain actions than it was before.Reward is the mechanism by which we select parameters, it is not something "given" to the model.
To (rather gruesomely) link this back to the dog analogy, RL is more like asking 100 dogs to sit, breeding the dogs which do sit and killing those which don't. Overtime, you will have a dog that can sit o...
