This document investigates how the quality of a reward model impacts the training efficiency of language models using Reinforcement Learning from Human Feedback (RLHF). It argues that while accuracy is typically used to assess reward models, it doesn't fully capture what makes a good "teacher." The key finding is that low reward variance, even with high accuracy, can lead to a flat objective landscape, significantly slowing down the optimization process. Furthermore, a reward model effective for one language model may not be for another, suggesting that evaluating reward models solely on accuracy or independent of the language model they guide is insufficient. Empirical evidence supports these theoretical findings.