This research explores Reinforcement Learning from Human Feedback (RLHF) under the KL-regularized contextual bandits framework. While traditional methods rely on complex optimistic or pessimistic estimates to manage uncertainty, the authors prove that greedy sampling—directly using empirical estimates—is surprisingly efficient. By leveraging the structural property that optimal policies remain within a bounded likelihood ratio of the reference policy, the study establishes logarithmic regret in online settings and optimal sample complexity for offline learning. These findings apply to both the Bradley-Terry reward-based model and general preference models, offering a more computationally efficient approach to aligning large language models. The theoretical results are further validated through simulations that show greedy sampling performs comparably to more sophisticated, resource-intensive algorithms.