This September 2025 paper introduces FlowRL, a novel reinforcement learning (RL) algorithm for large language models (LLMs) that shifts the optimization objective from reward maximization to reward distribution matching via flow balancing. Traditional RL methods like PPO and GRPO tend to over-optimize high-reward paths, leading to limited solution diversity and mode collapse, particularly in complex tasks like long Chain-of-Thought (CoT) reasoning. FlowRL addresses this by minimizing the reverse KL divergence between the policy and a reward-weighted target distribution, which is shown to be equivalent to the trajectory balance loss from GFlowNets, thereby jointly promoting reward and entropy maximization. Through experiments on math and code reasoning benchmarks, FlowRL demonstrates significant performance gains—an average of 10.0% over GRPO and 5.1% over PPO on math tasks—by generating substantially more diverse and generalizable reasoning trajectories.

Source:

https://arxiv.org/pdf/2509.15207