Neural intel Pod

This paper introduces a novel framework for offline reinforcement learning (RL), specifically addressing challenges in scenarios with continuous action spaces and unmeasured confounding variables. The authors develop a method for nonparametric estimation of policy value within an infinite-horizon framework by establishing a new identification result that utilizes "reward-inducing proxy variables." Based on this, they propose a minimax estimator and a policy-gradient-based algorithm to find optimal policies, providing theoretical guarantees for consistency and error bounds. The methodology's effectiveness is demonstrated through extensive simulations and a real-world application involving the German Family Panel data, aiming to identify optimal strategies for enhancing long-term relationship satisfaction.

This document outlines a novel deep reinforcement learning (DRL) framework for optimizing the placement of air purification booths in metropolitan areas, using Delhi, India as a case study. The research highlights the limitations of traditional pollution mitigation strategies and proposes a data-driven approach that integrates various urban factors like population density, traffic patterns, and industrial influence. By employing the Proximal Policy Optimization (PPO) algorithm, the framework aims to maximize air quality improvement (AQI) while ensuring equitable spatial coverage and adhering to practical constraints. The study compares this AI-driven solution against random and greedy placement methods, demonstrating its superiority in achieving a balanced and effective distribution of air purification infrastructure for smarter, healthier cities.

This academic paper introduces Non-Stationary Natural Actor-Critic (NS-NAC), a novel model-free, policy-based reinforcement learning algorithm designed for time-varying environments where rewards and transition probabilities change. Traditional reinforcement learning often assumes stationary environments, but real-world applications frequently involve dynamic systems. NS-NAC addresses this by incorporating restart-based exploration and adaptive learning rates to balance forgetting outdated information and learning new environmental dynamics. The paper also presents BORL-NS-NAC, a parameter-free extension that does not require prior knowledge of environmental variation, and provides theoretical guarantees for both algorithms through dynamic regret analysis, supported by empirical simulations.

This document introduces a novel framework for offline reinforcement learning (RL), focusing on optimizing individual policies when data comes from diverse or heterogeneous populations. The authors propose using individualized latent variables within a shared heterogeneous model to efficiently estimate unique Q-functions for each individual. Their Penalized Pessimistic Personalized Policy Learning (P4L) algorithm offers theoretical guarantees for a fast average regret rate under a weak partial coverage assumption. The research highlights the limitations of traditional RL methods that assume population homogeneity, which often lead to suboptimal policies for diverse groups. Simulation studies and a real-world application in intensive care demonstrate the superior performance of their proposed method compared to existing approaches.

This article from Amazon Science, published in May 2025, focuses on machine learning and conversational AI, specifically addressing improvements in reinforcement learning with human feedback (RLHF) for large language models (LLMs). The authors, Sailik Sengupta and Saket Dingliwal, introduce SeRA (self-reviewing and alignment), a novel training method designed to mitigate spurious correlations that can arise during direct preference optimization (DPO). SeRA refines the training process by prioritizing preference pairs with significant reward differences, which the researchers assert can boost performance by 20% to 40%. The text also highlights related Amazon Science work and outlines various applied scientist career opportunities within Amazon, emphasizing roles in generative AI and LLMsacross different departments like Prime Video and Alexa AI.

This document introduces AXIOM, a novel artificial intelligence architecture designed to learn how to play games efficiently using object-centric models and active inference. Unlike traditional deep reinforcement learning (DRL)methods that require vast amounts of data, AXIOM leverages inherent priors about objects and their interactions, enabling it to master various games within 10,000 interaction steps. The system employs a Slot Mixture Model (sMM) for visual parsing, an Identity Mixture Model (iMM) for object classification, a Transition Mixture Model (tMM) for dynamics, and a Recurrent Mixture Model (rMM) to infer dynamic transitions, all of which adapt their complexity by growing and pruning components. This approach significantly reduces computational costs and offers interpretable world models compared to state-of-the-art DRL baselines, demonstrating robustness to environmental changes like color and shape perturbations.

This academic paper explores a critical issue in reinforcement learning (RL) with large language models (LLMs): the rapid decline of policy entropy, which limits the models' ability to explore and improve. The authors demonstrate an empirical relationship where performance gains are directly tied to entropy reduction, leading to a predictable performance ceiling. To address this, they analyze the dynamics of policy entropy, showing its change is linked to the covariance between action probability and advantage. Based on this understanding, the paper proposes two novel techniques, Clip-Cov and KL-Cov, which effectively manage entropy by restricting updates to high-covariance tokens, thus promoting continuous exploration and achieving superior performance in reasoning tasks.

AI and Robotics

This academic paper explores ZeroTIR, a novel method for training Large Language Models (LLMs) to spontaneously use external tools, specifically Python code execution, for mathematical problem-solving through Reinforcement Learning (RL). The authors identify and characterize Agent RL Scaling Laws, demonstrating that as RL training progresses, there are predictable increases in code execution frequency, response length, and task accuracy. They propose and implement an efficient framework (ARL) for this training, showing that ZeroTIR models (ZTRL)significantly outperform traditional RL methods without tool integration on challenging math benchmarks. The research highlights that while increased interaction potential improves results, models tend to converge on strategies that favor fewer, high-utility code calls for successful solutions.

This academic paper critically re-evaluates the widespread belief that Reinforcement Learning with Verifiable Rewards (RLVR) enhances the fundamental reasoning capabilities of large language models (LLMs) beyond their initial base models. The authors employ the pass@k metric across various benchmarks, including mathematics, code generation, and visual reasoning, to assess the boundary of reasoning capacity by allowing models multiple attempts to solve problems. Surprisingly, the study finds that while RLVR training improves sampling efficiency (better performance at smaller k values), it does not introduce novel reasoning patterns; instead, the reasoning paths of RL-trained models are already present within the base models' output distributions, and RLVR even reduces the overall scope of solvable problems at larger k values. The research concludes that distillation, unlike RLVR, can genuinely introduce new knowledge and expand a model's reasoning boundary, suggesting a need for alternative training paradigms to truly advance LLM reasoning.