This paper examines the problem of aligning large language models (LLMs) with human preferences using Reinforcement Learning with Human Feedback (RLHF). The authors argue that the reliability of reward models, which are used to estimate human preferences, is a significant challenge in RLHF. They demonstrate that reward models trained on limited datasets with stochastic optimization algorithms can exhibit substantial variability, leading to uncertainty in the reward estimates. The paper proposes a variance-aware policy optimization method that accounts for this uncertainty by incorporating a weighted constraint based on the variance of reward estimates. Through theoretical analysis and experiments, the authors show that their proposed method effectively reduces the risk of policy degradation in scenarios with noisy reward models. The paper also presents empirical results on an ensemble of reward models trained on a large preference dataset, confirming the variability of reward estimates and demonstrating the efficacy of their variance-aware approach in improving the robustness and safety of aligned LLMs.