This research introduces A-PO*, a new reinforcement learning approach for refining large language models to enhance their reasoning capabilities. Unlike existing methods that are often computationally expensive and memory-intensive due to requiring multiple generations per prompt or explicit critic networks, A*-PO streamlines the process. It accomplishes this by initially estimating the optimal value function offline using samples from a reference policy, then performing on-policy updates with only a single response per prompt. The paper demonstrates that A*-PO achieves competitive performance while being significantly faster and more memory-efficient across various mathematical reasoning tasks and model sizes, supported by theoretical analysis and experimental results.