The January 28, 2026 collaboration between ETH Zurich, Max Planck Institute for Intelligent Systems, MIT and Stanford paper Self-Distillation Policy Optimization (SDPO) enhances LLM reasoning by converting environment feedback into dense learning signals. Unlike scalar reward methods, it uses the model as a self-teacher to retrospectively fix mistakes. This improves sample efficiency and accuracy at scale. Source: https://arxiv.org/pdf/2601.20802 Title: Reinforcement Learning via Self-Distillation January 28, 2026. Institutions: * ETH Zurich * Max Planck Institute for Intelligent Systems * MIT * Stanford Authors: * Jonas Hubotter (ETH Zurich) * Frederike Lubeck (ETH Zurich, Max Planck Institute for Intelligent Systems) * Lejs Behric (ETH Zurich) * Anton Baumann (ETH Zurich) * Marco Bagatella (ETH Zurich, Max Planck Institute for Intelligent Systems) * Daniel Marta (ETH Zurich) * Ido Hakimi (ETH Zurich) * Idan Shenfeld (MIT) * Thomas Kleine Buening (ETH Zurich) * Carlos Guestrin (Stanford) * Andreas Krause (ETH Zurich)

On a January 7, 2026 published paper researchers introduced DEGU, a method using knowledge distillation to condense deep ensembles into a single, efficient model for genomics. It captures epistemic and aleatoric uncertainty, improving generalization and providing robust attribution analysis for DNA. Source: January 07 2026 Uncertainty-aware genomic deep learning with knowledge distillation Simons Center for Quantitative Biology, Cold Spring Harbor Laboratory Jessica Zhou, Kaeli Rizzo, Trevor Christensen, Ziqi Tang, Peter K. Koo https://doi.org/10.1038/s44387-025-00053-3

The January 27, 2026 ByteDance paper "Post-LayerNorm Is Back: Stable, ExpressivE, and Deep" introduces th Keel architecture which addresses the optimization instability of deep Transformers by reviving the Post-LayerNorm formulation, which theoretically offers better expressivity than the standard Pre-LayerNorm but historically fails to train at scale due to gradient vanishing. By replacing the standard ResNet-style residual pathway with a Highway-style connection and injecting an additional normalization step into the residual branch, Keel preserves gradient magnitude and enables stable training at depths exceeding 1,000 layers without requiring complex initialization. Empirical evaluations show that Keel tolerates significantly higher learning rates and outperforms Pre-LayerNorm baselines in reasoning and coding tasks, proving that depth scaling remains a viable path for improving model performance. The Keel paper argues that extending context length is limited to expanding information access rather than improving "fundamental expressivity," asserting that while longer contexts allow models to process more data, they do not inherently unlock the capacity for complex hierarchical reasoning. Whereas Gemini 1.5 demonstrates that long context drives In-Context Learning (ICL) and reduces predictive uncertainty (NLL) via retrieval-heavy tasks like learning a language from a manual, Keel contends that this does not equate to the "qualitatively new behaviors" unlocked by depth scaling. Consequently, Keel frames depth as the superior, albeit historically unstable, axis for improving model reasoning (particularly in math and code), contrasting the "diminishing returns" and high cost of context scaling against the robust expressivity gains achieved by stabilizing deeper networks.Sources:Keel:https://arxiv.org/pdf/2601.19895The Gemini 1.5 paper with its power law:https://arxiv.org/pdf/2403.05530

There is a sharp divergence regarding the utility of long context. Google's Gemini 1.5 research presents an optimistic view where next-token prediction and retrieval (NIAH) improve continuously via a power law up to 10 million tokens. Broader research counters that while *retrieval* scales, utilitarian value (downstream task performance like reasoning or summarization) saturates rapidly or degrades due to "lost-in-the-middle" effects and data scarcity. There is no conclusive position on the empirical utility of long context for complex reasoning; the community must move beyond simple retrieval benchmarks to determine if the immense cost of processing millions of tokens yields proportional functional gains.Sources:1. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextDate: March 2024Institutions: Google DeepMindURL:https://arxiv.org/pdf/2403.055302. How to Train Long-Context Language Models (Effectively) [ProLong]Date: 2025Institutions: Princeton UniversityURL: https://aclanthology.org/2025.acl-long.366.pdf3. L2M: Mutual Information Scaling Law for Long-Context Language ModelingDate:** 2025 (NeurIPS)Institutions: MIT, Polytechnic University of Catalonia, Harvard University, UCLAURL:https://arxiv.org/pdf/2503.047254. Predicting Task Performance with Context-aware Scaling LawsDate: October 2025Institutions: UC Santa Cruz, Washington University in St. Louis, Databricks, Google DeepMind, UC BerkeleyURL:https://arxiv.org/pdf/2510.149195. Explaining Context Length Scaling and Bounds for Language ModelsDate:.February 2025Institutions:Tsinghua University, CPHOS Research, Carnegie Mellon University, University of Washington, University of CopenhagenURL:https://arxiv.org/pdf/2502.014816. Scaling Laws and In-Context Learning: A Unified Theoretical FrameworkDate: November 2025 (NeurIPS)URL: https://arxiv.org/pdf/2511.062327. Long-Context Efficient Transformers: A Comprehensive Survey of Techniques, Applications, and Future DirectionsDate: April 10, 2025Institutions: Tsinghua University, Peking University, USTC, Stanford University, UC BerkeleyURL:https://www.techrxiv.org/users/892385/articles/1283745-long-context-efficient-transformers-a-comprehensive-survey-of-techniques-applications-and-future-directions

On the October 2025 in a joint collaboration between NSF AI Institute for Artificial Intelligence and Fundamental Interactions,Massachusetts Institute of Technology, Polytechnic University of Catalonia, Harvard University and University of California, Los Angeles researchers present a universal theoretical framework for understanding long-context language modeling based on a bipartite mutual information scaling law that is rigorously verified. This is in the paper "L2M: Mutual Information Scaling Law for Long-Context Language Modeling". This research paper investigates how large language models manage long-range dependencies by applying principles from information theory. The authors introduce the L2M framework, which establishes that a model's ability to process extensive context is strictly limited by the size of its history state. While transformers utilize a growing cache of data to maintain performance across long sequences, models with fixed-size states, such as SSMs and RNNs, face inherent capacity bottlenecks as input length increases. By utilizing bipartite mutual information as a metric, the study formalizes the theoretical requirements for an architecture to be considered MI-capable. Ultimately, this work provides a principled method for evaluating and designing efficient architectures that can sustain complex reasoning over thousands of tokens. Source: https://arxiv.org/pdf/2503.04725

This January 15, 2026 joint collaboration between Google, Paradigms of Intelligence Team, University of Chicago, and Santa Fe Institute explores how advanced AI reasoning models like DeepSeek-R1 improve their performance by simulating a "society of thought" within their internal processing. Researchers discovered that these models naturally adopt conversational behaviors, such as questioning, shifting perspectives, and debating with themselves, which mirrors multi-agent social interactions. By using mechanistic interpretability to steer specific conversational features, the team demonstrated that these social simulations causally increase cognitive strategies like backtracking and verification. Furthermore, models specifically trained on multi-agent dialogues achieve higher accuracy faster than those using standard step-by-step reasoning. The study ultimately suggests that collective intelligence emerges internally through the competition and integration of diverse simulated personas. These findings indicate that the most effective machine reasoning is not a flat monologue, but a complex, socially structured dialogue. Source: https://arxiv.org/pdf/2601.10825

The paper, titled "Can Language Models Discover Scaling Laws?" and published on January 22, 2026, represents a collaborative effort by researchers from Peking University, Stanford University, Wizard Quant, and Tsinghua University. he authors address the inefficiency of manual scaling law discovery by introducing SLDAgent, an evolution-based system designed to automate the search for predictive symbolic formulas. They prove that this agent consistently discovers laws with superior extrapolation accuracy compared to human experts and existing baselines across the newly curated SLDBench, a testbed aggregating over 5,000 training experiments. Additionally, the study demonstrates the practical superiority of these autonomously discovered laws in critical tasks such as analytical hyperparameter optimization and pre-trained model selection. Source: https://arxiv.org/pdf/2507.21184

This January 18, 2026 massive collaboration between University of Illinois Urbana-Champaign, Meta, Amazon, Google Deepmind, UCSD and Yale explores the evolution of agentic reasoning in large language models, moving beyond static text generation toward dynamic planning and external interaction. It details how models utilize tool integration and multi-agent systems to solve complex problems in fields like software engineering, scientific discovery, and robotics. The text categorizes specialized roles within these systems—such as leaders, executors, and critics—to facilitate sophisticated collaboration and feedback-driven behaviors. Furthermore, it examines various post-training methods and architectural frameworks designed to optimize how agents communicate and manage long-term memory. Finally, the sources provide a comprehensive overview of benchmarks used to evaluate these autonomous systems across diverse, real-world applications. Source: January 18, 2026 Agentic Reasoning for Large Language Models https://arxiv.org/pdf/2601.12538

OpenAI manages a massive PostgreSQL infrastructure to support hundreds of millions of users by utilizing a single-primary architecture with dozens of global read replicas. To maintain stability under extreme traffic, the engineering team implemented rigorous query optimizations, connection pooling through PgBouncer, and aggressive caching strategies. They mitigate the limitations of a single writer by migrating write-heavy workloads to sharded systems like Azure Cosmos DB and enforcing strict rate limits. High availability is ensured through regional workload isolation and the use of hot standbys to prevent total service outages. This technical evolution allows the platform to process millions of queries per second while maintaining low latency and high reliability. Future scaling plans include testing cascading replication to expand their global database footprint even further. Source: https://openai.com/index/scaling-postgresql/

On January 14, 2026 Sequoia Capital published a piece assertion that Artificial General Intelligence has arrived ahead of schedule, redefined as the functional ability for AI to "figure things out" autonomously. The authors highlight the transition from simple conversational models to long-horizon agents that can persist through complex, multi-step tasks without constant human guidance. By leveraging reasoning capabilities and iterative problem-solving, these agents are now performing specialized work in fields like coding, recruiting, and law. The source predicts an exponential growth in agent performance, suggesting that AI will soon manage workloads that would take human experts years to complete. Consequently, founders are encouraged to pivot from building chatbots to developing autonomous colleagues that sell completed outcomes rather than just software. This shift marks a new era where AI moves from being a passive tool to a proactive doer capable of navigating real-world ambiguity. Source: https://sequoiacap.com/article/2026-this-is-agi/