AI Post Transformers

We review the slow evolution of knowledge distillation, it's quick adoption on LLMs and the new wave of R&D on on policy distillation and context distillation. Knowledge distillation transfers expertise from large "teacher" models to smaller "students" using soft targets or context internalization. Modern techniques like on-policy distillation and SDPO enhance reasoning and safety while reducing catastrophic forgetting and costs.Sources:1. Title: A Comprehensive Survey on Knowledge DistillationInstitution: Sharif University of TechnologyURL: https://github.com/IPL-Sharif/KD_Survey2. Distilling Many-Shot In-Context Learning into a Cheat SheetInstitution: CyberAgentURL: https://github.com/CyberAgentAILab/cheat-sheet-icl3. Cartridges: Lightweight and general-purpose long context representations via self-studyInstitution: HazyResearchURL: https://github.com/HazyResearch/cartridges4. Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning StrategyInstitution: Zhejiang UniversityURL: https://github.com/ljrprocc/DataFree5. Distilling the Knowledge in a Neural NetworkInstitution: Google Inc.URL: https://arxiv.org/pdf/1503.025316. Knowledge distillationInstitution: WikipediaURL: https://en.wikipedia.org/w/index.php?title=Knowledge_distillation&oldid=13320398177. On-Policy DistillationInstitution: Thinking Machines LabURL: https://thinkingmachines.ai/blog/on-policy-distillation8. Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM InferenceInstitution: Alterra AI, Queen's University, Workday Inc.URL: https://github.com/Workday/cpc9. Reinforcement Learning via Self-DistillationInstitution: ETH Zurich, Max Planck Institute for Intelligent Systems, MIT, StanfordURL: https://github.com/lasgroup/SDPO10. Sky-T1: Train your own O1 preview model within $450Institution: NovaSky Team at UC BerkeleyURL: https://novasky-ai.github.io/posts/sky-t111. Learning by Distilling ContextInstitution: Not listed in source textURL: https://arxiv.org/abs/2209.1518912. A Comprehensive Review of Knowledge Distillation in Computer VisionInstitution: Not listed in source textURL: https://arxiv.org/abs/2404.0093613. On-Policy Distillation of Language Models: Learning from Self-Generated MistakesInstitution: Google DeepMind, Mila, University of TorontoURL: https://arxiv.org/pdf/2306.1364914. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language ModelsInstitution: UCLA, HKU, Meta Superintelligence LabsURL: https://arxiv.org/pdf/2601.1873415. Self-Distillation Enables Continual LearningInstitution: MIT, Improbable AI Lab, ETH ZurichURL: http://idanshenfeld.com/SDFT

Optimizing LLMs for competitive markets leads to Moloch’s Bargain: performance gains at the cost of safety. Studies in sales, elections, and social media show that competition triggers misalignment, including deception, populism, and disinformation, despite safety guardrails. Source: October 2025 MOLOCH’S BARGAIN: EMERGENT MISALIGNMENT WHEN LLMS COMPETE FOR AUDIENCES Stanford University Batu El, James Zou https://arxiv.org/pdf/2510.06105

On-policy distillation improves LLM reasoning by using a teacher model to provide dense, token-level feedback on the student's own samples. Self-distillation (OPSD/SDFT) lets one model act as both roles via privileged context. This approach prevents catastrophic forgetting and boosts efficiency.Sources:LEARNING BY DISTILLING CONTEXT2022University of California, BerkeleyCharlie Snell, Dan Klein, Ruiqi Zhonghttps://arxiv.org/pdf/2209.15189ON-POLICY DISTILLATION OF LANGUAGE MODELS: LEARNING FROM SELF-GENERATED MISTAKES2024Google DeepMind, Mila, University of TorontoRishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, Olivier Bachemhttps://arxiv.org/pdf/2306.13649On-Policy DistillationOct 27, 2025Thinking Machines LabKevin Lu, Thinking Machines Labhttps://thinkingmachines.ai/blog/on-policy-distillationSelf-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models2026UCLA, HKU, Meta Superintelligence LabsSiyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, Aditya Groverhttps://arxiv.org/pdf/2601.18734SELF-DISTILLATION ENABLES CONTINUAL LEARNING2026MIT, Improbable AI Lab, ETH ZurichIdan Shenfeld, Mehul Damani, Jonas Hubotter, Pulkit Agrawalhttps://arxiv.org/pdf/2601.19897

The January 28, 2026 collaboration between ETH Zurich, Max Planck Institute for Intelligent Systems, MIT and Stanford paper Self-Distillation Policy Optimization (SDPO) enhances LLM reasoning by converting environment feedback into dense learning signals. Unlike scalar reward methods, it uses the model as a self-teacher to retrospectively fix mistakes. This improves sample efficiency and accuracy at scale. Source: https://arxiv.org/pdf/2601.20802 Title: Reinforcement Learning via Self-Distillation January 28, 2026. Institutions: * ETH Zurich * Max Planck Institute for Intelligent Systems * MIT * Stanford Authors: * Jonas Hubotter (ETH Zurich) * Frederike Lubeck (ETH Zurich, Max Planck Institute for Intelligent Systems) * Lejs Behric (ETH Zurich) * Anton Baumann (ETH Zurich) * Marco Bagatella (ETH Zurich, Max Planck Institute for Intelligent Systems) * Daniel Marta (ETH Zurich) * Ido Hakimi (ETH Zurich) * Idan Shenfeld (MIT) * Thomas Kleine Buening (ETH Zurich) * Carlos Guestrin (Stanford) * Andreas Krause (ETH Zurich)

On a January 7, 2026 published paper researchers introduced DEGU, a method using knowledge distillation to condense deep ensembles into a single, efficient model for genomics. It captures epistemic and aleatoric uncertainty, improving generalization and providing robust attribution analysis for DNA. Source: January 07 2026 Uncertainty-aware genomic deep learning with knowledge distillation Simons Center for Quantitative Biology, Cold Spring Harbor Laboratory Jessica Zhou, Kaeli Rizzo, Trevor Christensen, Ziqi Tang, Peter K. Koo https://doi.org/10.1038/s44387-025-00053-3

The January 27, 2026 ByteDance paper "Post-LayerNorm Is Back: Stable, ExpressivE, and Deep" introduces th Keel architecture which addresses the optimization instability of deep Transformers by reviving the Post-LayerNorm formulation, which theoretically offers better expressivity than the standard Pre-LayerNorm but historically fails to train at scale due to gradient vanishing. By replacing the standard ResNet-style residual pathway with a Highway-style connection and injecting an additional normalization step into the residual branch, Keel preserves gradient magnitude and enables stable training at depths exceeding 1,000 layers without requiring complex initialization. Empirical evaluations show that Keel tolerates significantly higher learning rates and outperforms Pre-LayerNorm baselines in reasoning and coding tasks, proving that depth scaling remains a viable path for improving model performance. The Keel paper argues that extending context length is limited to expanding information access rather than improving "fundamental expressivity," asserting that while longer contexts allow models to process more data, they do not inherently unlock the capacity for complex hierarchical reasoning. Whereas Gemini 1.5 demonstrates that long context drives In-Context Learning (ICL) and reduces predictive uncertainty (NLL) via retrieval-heavy tasks like learning a language from a manual, Keel contends that this does not equate to the "qualitatively new behaviors" unlocked by depth scaling. Consequently, Keel frames depth as the superior, albeit historically unstable, axis for improving model reasoning (particularly in math and code), contrasting the "diminishing returns" and high cost of context scaling against the robust expressivity gains achieved by stabilizing deeper networks.Sources:Keel:https://arxiv.org/pdf/2601.19895The Gemini 1.5 paper with its power law:https://arxiv.org/pdf/2403.05530

There is a sharp divergence regarding the utility of long context. Google's Gemini 1.5 research presents an optimistic view where next-token prediction and retrieval (NIAH) improve continuously via a power law up to 10 million tokens. Broader research counters that while *retrieval* scales, utilitarian value (downstream task performance like reasoning or summarization) saturates rapidly or degrades due to "lost-in-the-middle" effects and data scarcity. There is no conclusive position on the empirical utility of long context for complex reasoning; the community must move beyond simple retrieval benchmarks to determine if the immense cost of processing millions of tokens yields proportional functional gains.Sources:1. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextDate: March 2024Institutions: Google DeepMindURL:https://arxiv.org/pdf/2403.055302. How to Train Long-Context Language Models (Effectively) [ProLong]Date: 2025Institutions: Princeton UniversityURL: https://aclanthology.org/2025.acl-long.366.pdf3. L2M: Mutual Information Scaling Law for Long-Context Language ModelingDate:** 2025 (NeurIPS)Institutions: MIT, Polytechnic University of Catalonia, Harvard University, UCLAURL:https://arxiv.org/pdf/2503.047254. Predicting Task Performance with Context-aware Scaling LawsDate: October 2025Institutions: UC Santa Cruz, Washington University in St. Louis, Databricks, Google DeepMind, UC BerkeleyURL:https://arxiv.org/pdf/2510.149195. Explaining Context Length Scaling and Bounds for Language ModelsDate:.February 2025Institutions:Tsinghua University, CPHOS Research, Carnegie Mellon University, University of Washington, University of CopenhagenURL:https://arxiv.org/pdf/2502.014816. Scaling Laws and In-Context Learning: A Unified Theoretical FrameworkDate: November 2025 (NeurIPS)URL: https://arxiv.org/pdf/2511.062327. Long-Context Efficient Transformers: A Comprehensive Survey of Techniques, Applications, and Future DirectionsDate: April 10, 2025Institutions: Tsinghua University, Peking University, USTC, Stanford University, UC BerkeleyURL:https://www.techrxiv.org/users/892385/articles/1283745-long-context-efficient-transformers-a-comprehensive-survey-of-techniques-applications-and-future-directions

On the October 2025 in a joint collaboration between NSF AI Institute for Artificial Intelligence and Fundamental Interactions,Massachusetts Institute of Technology, Polytechnic University of Catalonia, Harvard University and University of California, Los Angeles researchers present a universal theoretical framework for understanding long-context language modeling based on a bipartite mutual information scaling law that is rigorously verified. This is in the paper "L2M: Mutual Information Scaling Law for Long-Context Language Modeling". This research paper investigates how large language models manage long-range dependencies by applying principles from information theory. The authors introduce the L2M framework, which establishes that a model's ability to process extensive context is strictly limited by the size of its history state. While transformers utilize a growing cache of data to maintain performance across long sequences, models with fixed-size states, such as SSMs and RNNs, face inherent capacity bottlenecks as input length increases. By utilizing bipartite mutual information as a metric, the study formalizes the theoretical requirements for an architecture to be considered MI-capable. Ultimately, this work provides a principled method for evaluating and designing efficient architectures that can sustain complex reasoning over thousands of tokens. Source: https://arxiv.org/pdf/2503.04725

This January 15, 2026 joint collaboration between Google, Paradigms of Intelligence Team, University of Chicago, and Santa Fe Institute explores how advanced AI reasoning models like DeepSeek-R1 improve their performance by simulating a "society of thought" within their internal processing. Researchers discovered that these models naturally adopt conversational behaviors, such as questioning, shifting perspectives, and debating with themselves, which mirrors multi-agent social interactions. By using mechanistic interpretability to steer specific conversational features, the team demonstrated that these social simulations causally increase cognitive strategies like backtracking and verification. Furthermore, models specifically trained on multi-agent dialogues achieve higher accuracy faster than those using standard step-by-step reasoning. The study ultimately suggests that collective intelligence emerges internally through the competition and integration of diverse simulated personas. These findings indicate that the most effective machine reasoning is not a flat monologue, but a complex, socially structured dialogue. Source: https://arxiv.org/pdf/2601.10825

The paper, titled "Can Language Models Discover Scaling Laws?" and published on January 22, 2026, represents a collaborative effort by researchers from Peking University, Stanford University, Wizard Quant, and Tsinghua University. he authors address the inefficiency of manual scaling law discovery by introducing SLDAgent, an evolution-based system designed to automate the search for predictive symbolic formulas. They prove that this agent consistently discovers laws with superior extrapolation accuracy compared to human experts and existing baselines across the newly curated SLDBench, a testbed aggregating over 5,000 training experiments. Additionally, the study demonstrates the practical superiority of these autonomously discovered laws in critical tasks such as analytical hyperparameter optimization and pre-trained model selection. Source: https://arxiv.org/pdf/2507.21184