AI Post Transformers

This September 2025 paper introduces FlowRL, a novel reinforcement learning (RL) algorithm for large language models (LLMs) that shifts the optimization objective from reward maximization to reward distribution matching via flow balancing. Traditional RL methods like PPO and GRPO tend to over-optimize high-reward paths, leading to limited solution diversity and mode collapse, particularly in complex tasks like long Chain-of-Thought (CoT) reasoning. FlowRL addresses this by minimizing the reverse KL divergence between the policy and a reward-weighted target distribution, which is shown to be equivalent to the trajectory balance loss from GFlowNets, thereby jointly promoting reward and entropy maximization. Through experiments on math and code reasoning benchmarks, FlowRL demonstrates significant performance gains—an average of 10.0% over GRPO and 5.1% over PPO on math tasks—by generating substantially more diverse and generalizable reasoning trajectories. Source: https://arxiv.org/pdf/2509.15207

This September 2025 paper introduces SearchInstruct, a novel framework designed to enhance Supervised Fine-Tuning (SFT) of large language models (LLMs) by constructing high-quality, domain-specific instruction datasets. This approach overcomes challenges like data scarcity and outdated model knowledge by dynamically retrieving external, up-to-date documents to generate accurate, context-grounded answers for augmented questions. SearchInstruct operates via a four-stage pipeline: starting with a small set of human-generated questions, expanding them using an LLM, retrieving relevant documents (via RAG or web search), and synthesizing context-aware responses. Experimental results confirm that the method significantly boosts LLM performance in specialized domains like Iranian culture and efficiently facilitates model editing for factual updates, creating a measurable improvement over baseline models. Source: https://arxiv.org/pdf/2509.10708

This September 2025 paper introduces Single-stream Policy Optimization (SPO), a new reinforcement learning algorithm for training Large Language Models (LLMs) developed by Tencent researchers. SPO challenges the prevailing group-based optimization methods like Group Relative Policy Optimization (GRPO), which suffer from high computational waste due to "degenerate groups" and synchronization bottlenecks, particularly in complex agentic tasks. The core of SPO involves returning to a single-stream paradigm, using a persistent, KL-adaptive value tracker as a stable baseline, and applying global advantage normalization to ensure efficient and stable learning. Empirical results on challenging math benchmarks, using the Qwen3-8B model, demonstrate that SPO consistently outperforms GRPO in terms of accuracy and achieves a significant 4.35x speedup in simulated high-variance agentic training environments, validating its superior scalability and efficiency. Source: https://arxiv.org/pdf/2509.13232

The "Anthropic Economic Index report" documents the rapid and uneven adoption of Artificial Intelligence (AI), specifically using data from the company's Claude.ai consumer platform and its enterprise API. The report highlights that AI adoption is geographically concentrated in high-income regions and that the speed of adoption surpasses previous technologies like the internet. Analyzing usage patterns, the authors find that coding tasks dominate usage across both consumer and enterprise users, but enterprise use via API is highly automated (77%) compared to the more collaborative (augmentation) consumer use. The study introduces the Anthropic AI Usage Index (AUI) to compare per-capita AI use across regions and suggests that for businesses, model capability and economic value matter more than cost, while the availability of contextual information may be a crucial bottleneck for sophisticated deployment. Source: https://assets.anthropic.com/m/218c82b858610fac/original/Economic-Index.pdf

This September 2025 paper describes THOR (Tool-Integrated Hierarchical Optimization via RL), a novel approach designed to enhance the mathematical reasoning and code generation capabilities of Large Language Models (LLMs) by integrating external code-execution tools. The methodology introduces TIRGen, a pipeline for creating high-quality Tool-Integrated Reasoning (TIR) data, which is crucial for training the model using a hierarchical reinforcement learning (RL) strategy. This RL framework incorporates both trajectory-level optimization for overall problem-solving ability and step-level optimization to correct code generation errors, addressing the sparse reward problem common in long reasoning tasks. Experimental results demonstrate that THOR achieves state-of-the-art (SOTA) performance across various mathematical and code benchmarks for both reasoning and non-reasoning models. Finally, the system leverages code execution feedback for a self-correction inference enhancement mechanism, which further improves performance, especially on more challenging problems. Source: https://arxiv.org/pdf/2509.13761

This paper published on Nature on September 17 2025, "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning," details the development of DeepSeek-R1-Zero and DeepSeek-R1, two large language models (LLMs) engineered to enhance reasoning capabilities. The authors explain how reinforcement learning (RL) is used to enable emergent advanced reasoning patterns like self-reflection and dynamic strategy adaptation, moving beyond reliance on human-annotated data. The paper discusses a multistage training pipeline for DeepSeek-R1, integrating rejection sampling, RL, and supervised fine-tuning to improve both reasoning and general language tasks while addressing issues like language mixing. Furthermore, the researchers highlight the release of these models and their distilled, smaller versions to the public to contribute to ongoing AI research. Ultimately, the source concludes by acknowledging the ethical considerations and limitations of their pure RL methodology, such as reward hacking and token efficiency. Source: https://www.nature.com/articles/s41586-025-09422-z

How can pre-computing and reusing Key-Value (KV) caches accelerate inference for Retrieval-Augmented Generation and other long-context LLM tasks? The provided sources identify the same core problem—high latency in Large Language Model (LLM) inference due to processing long, repetitive contexts—and converge on a unified solution: leveraging pre-computed Key-Value (KV) caches. Each source then contributes a unique perspective on *how* to implement this solution effectively, addressing specific challenges that arise from this approach. The unified answer proposed by all sources is to avoid redundant computation by pre-computing, storing, and reusing the KV caches of recurring text segments (referred to as chunks, documents, or prompt modules).Sources:https://arxiv.org/html/2502.15734v1https://arxiv.org/html/2412.15605v1https://arxiv.org/html/2502.16002v1https://arxiv.org/html/2310.07240v6https://arxiv.org/pdf/2404.12457https://openreview.net/pdf?id=x7NbaU8RSUhttps://proceedings.mlsys.org/paper_files/paper/2024/file/a66caa1703fe34705a4368c3014c1966-Paper-Conference.pdfhttps://www.cs.princeton.edu/~ravian/COS597_F24/papers/cacheblend.pdf

This September 2025 academic paper, titled "REFRAG: Rethinking RAG based Decoding," appears on the alphaXiv pre-print server. It focuses on Reframing Retrieval-Augmented Generation (RAG) within the context of decoding processes. The research likely explores new approaches or improvements to how RAG mechanisms are integrated and utilized during the generation of text. By "rethinking" this integration, the authors probably aim to optimize the performance or efficiency of language models that leverage external knowledge through RAG. The source, specifically from alphaXiv, indicates it is a scholarly work shared for peer review and discussion within the scientific community. Source: https://www.alphaxiv.org/abs/2509.01092v1

This July 2025 paper introduces Hierarchical Networks (H-Nets), a novel architecture designed to move beyond traditional tokenization in large language models by implementing dynamic chunking. This mechanism allows the model to automatically learn content- and context-dependent segmentation strategies directly from raw data, eliminating the need for predefined pre-processing steps like byte-pair encoding (BPE). H-Nets utilize a recursive, multi-stage structure that processes data at varying levels of abstraction, from bytes to more complex semantic units. Experiments demonstrate that H-Nets, particularly multi-stage configurations, outperform tokenized Transformers in perplexity, downstream tasks, and robustness to textual perturbations, especially in languages and modalities with weak or absent tokenization cues, such as Chinese, code, and DNA sequences. The authors highlight that this end-to-end learning of data chunking represents a significant step towards more generalized and efficient foundation models. Source: https://arxiv.org/html/2507.07955

This April 2025 paper introduces Infini-gram, a novel engine designed to scale n-gram language models to an unprecedented 5 trillion tokens and support unbounded n (∞-gram LMs). Unlike traditional methods that rely on pre-computed count tables, Infini-gram leverages suffix arrays for efficient, low-latency calculation of n-gram and ∞-gram probabilities, even for extremely long contexts. The authors demonstrate that this modernized approach significantly improves the perplexity of neural Large Language Models (LLMs), by up to 73%, by offering complementary insights into human-written and machine-generated text. Beyond enhancing LLMs, the Infini-gram engine also enables various applications such as corpus analysis, data curation, document retrieval, and detection of data contamination. A public web interface, API endpoint, and source code are provided to encourage further exploration and development within the community. Source: https://arxiv.org/pdf/2401.17377