This research introduces **Self-Distillation Fine-Tuning (SDFT)**, a novel on-policy learning method designed to help large language models acquire new skills without suffering from **catastrophic forgetting**. Unlike traditional supervised fine-tuning, which often causes models to lose prior knowledge, **SDFT** utilizes the model’s own **in-context learning** abilities by using a version of itself conditioned on demonstrations as a teacher. This approach generates **on-policy training signals** that allow the model to internalize new facts and reasoning patterns while remaining close to its original parameter distribution. Empirical results across **skill acquisition** and **knowledge injection** tasks show that **SDFT** consistently outperforms existing baselines in both task accuracy and the preservation of general capabilities. Ultimately, the research positions **self-distillation** as a practical and scalable path for enabling **continual learning** in foundation models.

This paper introduces **Maximum Likelihood Reinforcement Learning (MaxRL)**, a novel framework designed to improve the training of models in tasks with binary feedback, such as mathematical reasoning and code generation. The authors argue that traditional **Reinforcement Learning (RL)** only optimizes a first-order approximation of the **maximum likelihood objective**, causing it to ignore harder problems where success is rare. **MaxRL** bridges this gap by using a compute-indexed objective that approaches exact maximum likelihood as more sampling resources are applied. By normalizing gradients based on successful outcomes rather than total samples, the method places greater emphasis on difficult tasks. Empirical results show that **MaxRL** significantly outperforms existing methods like **GRPO**, offering superior scaling with data and up to **20x gains in inference efficiency**. Ultimately, the framework mitigates the "distribution sharpening" and diversity loss often seen in large reasoning models trained with standard RL.

This research demonstrates that fixed-weight Transformers can function as versatile algorithm emulators by simply modifying the input prompt. The authors prove that a minimal attention architecture can execute a wide variety of machine learning tasks, such as gradient descent and linear regression, without updating its internal parameters. They distinguish between task-specific emulation, where a dedicated module performs one routine, and a more powerful prompt-programmable mode where a single module hosts a library of different algorithms. This capability is achieved by encoding algorithmic instructions and parameters directly into the prompt's tokens, allowing the model to swap routines on the fly. Mathematical proofs and experiments confirm that softmax attention alone is sufficient to achieve this algorithmic universality. Ultimately, the study provides a theoretical foundation for understanding how foundation models like GPT can adapt to complex new tasks through context alone.

This research paper introduces PPI-SVRG, a novel optimization framework designed for semi-supervised learning when labeled data is limited but machine learning predictions are plentiful. The authors prove that two popular statistical techniques—Prediction-Powered Inference (PPI) and Stochastic Variance Reduced Gradient (SVRG)—share a mathematical foundation based on control variates. By merging these methods, the new algorithm uses abundant unlabeled data and pre-trained model predictions to stabilize gradients and reduce variance. The study provides convergence guarantees showing that while poor predictions might create an error floor, they do not jeopardize the overall stability of the optimization process. Empirical tests demonstrate significant gains, including a 43–52% reduction in mean squared error and improved accuracy on image classification tasks. Ultimately, the work offers a robust way to accelerate model training by effectively leveraging cheap, automated predictions to supplement expensive human-labeled information.

This research explores model collapse, a phenomenon where generative models degrade after being repeatedly trained on their own synthetic outputs. The authors provide a theoretical framework using Maximum Likelihood Estimation (MLE) to determine when this process can be avoided. They demonstrate that if models meet specific regularity and smoothness assumptions, they can remain consistent and accurate even as the proportion of real data diminishes. Conversely, the study provides the first rigorous proof that without these structural assumptions, model collapse can occur abruptly or over time, even when real data is preserved. Ultimately, the findings suggest that data accumulation alone does not guarantee stability; rather, the underlying mathematical properties of the distribution family are what prevent performance failure.

This paper introduces the DRQ-learner, a novel causal inference meta-learner designed to predict individualized outcomes in Markov Decision Processes (MDPs). While traditional methods often struggle with the "curse of horizon" or lack theoretical stability, this new approach provides a foundation for more reliable personalized medicine and sequential decision-making. The authors leverage statistical orthogonality to ensure the model remains robust against errors in secondary estimation tasks and model misspecification. Through its doubly robust and quasi-oracle efficient properties, the learner performs as effectively as if the true underlying data distributions were already known. Empirical tests in simulated environments confirm that the DRQ-learner outperforms existing baselines, particularly in complex scenarios with low data overlap and long-term horizons. Ultimately, the research bridges the gap between causal treatment effect estimation and reinforcement learning to enhance patient-specific therapeutic strategies.

This paper explores pretraining data filtering as a robust strategy for shaping the capabilities of large language models, specifically by selectively removing undesired knowledge like medical or hazardous information. Research indicates that token-level filtering is more precise and efficient than document-level approaches, allowing models to retain general performance while significantly increasing the difficulty for adversaries to recover suppressed traits. As pretraining compute scales, this method becomes exponentially more effective, resulting in a 7000x compute slowdown for those attempting to relearn the "forgotten" domain. Furthermore, models trained via this method remain corrigible and easier to align, debunking concerns that removing data makes them harder to control. The authors also introduce a scalable pipeline using sparse autoencoders to generate high-quality labels from weak or noisy supervision. Ultimately, the study advocates for intervention during pretraining as a foundational, tamper-resistant layer for AI safety and security.

Researchers from Meta’s FAIR division introduced Self-Improving Pretraining, a novel framework that enhances large language models by integrating reinforcement learning and post-trained judges directly into the pretraining phase. Unlike standard next-token prediction, this method streams data and uses an existing high-quality model to rewrite suffixes and evaluate multiple model rollouts for quality, safety, and truthfulness. This approach ensures that core behaviors like factuality and safety are established from the start, rather than being treated as secondary corrections during fine-tuning. Experimental results demonstrate significant improvements, including a 36.2% increase in factuality and an 18.5% boost in safety compared to traditional baselines. Ultimately, the system allows models to learn how to steer away from low-quality content by rewarding superior generation candidates during the initial learning process.

This paper provides a formal theoretical framework for success conditioning, a widely used reinforcement learning heuristic employed in Decision Transformers and language model alignment. The author proves that this technique is not merely a heuristic but exactly solves a trust-region optimization problem using a unique chi-squared divergence constraint. A central contribution is the Action-Influence Identity, which demonstrates that the magnitude of policy improvement is equal to the statistical variability in success rates attributable to the behavior policy's actions. This identity reveals that success conditioning is inherently conservative: it avoids dangerous distribution shifts by design and fails only when it becomes overly cautious in the absence of sufficient signal. Furthermore, the research explains how return thresholding acts as a proxy that can amplify these improvements, provided the chosen success criteria remain aligned with the true objective. Ultimately, the work bridges the gap between simple supervised fine-tuning on successful outcomes and the rigorous mathematical foundations of policy optimization.

This paper introduces Trajectory Bellman Residual Minimization (TBRM), a new value-based reinforcement learning algorithm designed to improve the reasoning capabilities of large language models. Unlike traditional policy-based methods like PPO or GRPO, TBRM optimizes a single trajectory-level objective using the model's own raw outputs as Q-values. This streamlined approach removes the need for complex components like critic models, importance sampling, or clipping, significantly reducing computational and memory overhead. The authors provide a theoretical proof of convergence to an optimal policy even when using arbitrary off-policy data in deterministic environments. Empirical tests on mathematical reasoning benchmarks show that TBRM matches or exceeds the performance of established baselines while being faster and more resource-efficient. Ultimately, the research suggests that value-based RL is a principled and powerful alternative for training models to handle complex, multi-step thinking tasks.