Sign up to save your podcasts
Or
Share 【第63期】无论DPO还是PPO，Preference Feedback应该怎么用？
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
【第63期】无论DPO还是PPO，Preference Feedback应该怎么用？
Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。
今天的主题是：Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference FeedbackSummary
This NeurIPS 2024 paper investigates the effectiveness of different components in preference-based learning for language models. The authors systematically compare Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) algorithms, examining the influence of preference data quality, reward model design, and policy training prompts on model performance across various benchmarks. Their findings highlight the importance of high-quality preference data and reveal that PPO generally outperforms DPO, though improvements from enhanced reward models are surprisingly limited. The researchers propose a recipe for effective preference-based learning and publicly release their code and datasets to promote further research in this area.
原文链接：https://arxiv.org/abs/2406.09279
View all episodes
By 任雨山Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。
今天的主题是：Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference FeedbackSummary
This NeurIPS 2024 paper investigates the effectiveness of different components in preference-based learning for language models. The authors systematically compare Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) algorithms, examining the influence of preference data quality, reward model design, and policy training prompts on model performance across various benchmarks. Their findings highlight the importance of high-quality preference data and reveal that PPO generally outperforms DPO, though improvements from enhanced reward models are surprisingly limited. The researchers propose a recipe for effective preference-based learning and publicly release their code and datasets to promote further research in this area.
原文链接：https://arxiv.org/abs/2406.09279