Paper Link: https://arxiv.org/abs/2604.12632

Summary:

The provided sources introduce Calibration-Aware Policy Optimization (CAPO), a new reinforcement learning framework designed to improve both the accuracy and reliability of Large Language Models (LLMs). The research identifies a critical flaw in existing methods like Group Relative Policy Optimization (GRPO), which frequently causes models to become overconfident in incorrect answers, a phenomenon known as calibration degradation. By implementing an uncertainty-aware advantage estimation based on a consistent logistic surrogate loss, CAPO ensures that model confidence aligns more accurately with factual correctness. The method also incorporates a reference-model-based noise masking mechanism to filter out low-quality training data, such as lucky guesses or near-correct reasoning. Extensive experiments across multiple mathematical reasoning benchmarks demonstrate that CAPO significantly reduces hallucinations and boosts inference-time performance. Ultimately, the sources highlight CAPO as a state-of-the-art advancement in creating trustworthy AI systems that better understand the limits of their own knowledge.