AI Post Transformers

This December 31, 2025 NVIDIA research introduces TTT-E2E, a novel approach to large language model memory that treats long-context processing as a continual learning problem rather than a structural design challenge. By utilizing test-time training, the model effectively compresses context into its own weights through next-token prediction, allowing it to adapt and learn while processing new information. Unlike traditional Transformers that suffer from linear latency growth, or Recurrent Neural Networks that experience performance loss at scale, TTT-E2E maintains constant inference speed without sacrificing accuracy. The method employs meta-learning during the pre-training phase to optimize the model’s initialization for these rapid weight updates at test time. Experimental results demonstrate that TTT-E2E achieves a 35x speedup over full attention at extreme context lengths while matching its scaling efficiency. Ultimately, the authors propose this end-to-end formulation as a fundamental solution to the computational bottlenecks of processing massive datasets.Sources:https://arxiv.org/pdf/2512.23675https://developer.nvidia.com/blog/reimagining-llm-memory-using-context-as-training-data-unlocks-models-that-learn-at-test-time/

Recent advancements in Long Context Language Models (LCLMs) demonstrate that In-Context Learning (ICL) capabilities follow predictable power-law scaling relationships, where performance improves monotonically with context length up to 10 million tokens and is governed by model depth, width, and training data volume. While Gemini 1.5 exhibits near-perfect recall and continued log-loss improvement at extreme scales, theoretical frameworks reveal that ICL functions mechanistically as implicit gradient descent, effectively performing low-rank weight updates to the model's MLP layers during inference. Furthermore, as context capacity expands, the necessity for sophisticated example selection strategies diminishes; simple random selection combined with data augmentation to fill the context window often yields optimal results, marking a shift from selection optimization to capacity utilization.Sources:1. **Gemini Team, Google** (2024) *Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context* https://arxiv.org/pdf/2403.055302. **Jinheon Baek, Sun Jae Lee, Prakhar Gupta, Geunseob (GS) Oh, Siddharth Dalmia, Prateek Kolhar** (2024) *Revisiting In-Context Learning with Long Context Language Models* https://arxiv.org/pdf/2412.169263. **Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, et al.** (2025) *A Comprehensive Survey on Long Context Language Modeling* https://arxiv.org/pdf/2503.174074. **Benoit Dherin, Michael Munn, Hanna Mazzawi, Michael Wunder, Javier Gonzalvo** (2025) *Learning without training: The implicit dynamics of in-context learning* https://arxiv.org/pdf/2507.160035. **Sushant Mehta, Ishan Gupta** (2025) *Scaling Laws and In-Context Learning: A Unified Theoretical Framework* https://arxiv.org/pdf/2511.06232

We focus on the July 2025 paper, "Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator". The paper goes into the mathematical details of approximating the FIM, and the easiest is what they call the Squisher: the Adam optimizer variance. This technique allows for tasks like model pruning and model merging to be performed "for free" without the significant computational overhead typically required to calculate the Fisher Information Matrix. We also review the old 1992 paper "Second order derivatives for network pruning: Optimal Brain Surgeon" in terms of what was missing in light of the Squisher paper.Sources:https://arxiv.org/pdf/2507.18807https://proceedings.neurips.cc/paper/1992/file/303ed4c69846ab36c2904d3ba8573050-Paper.pdf

On January 12, 2026 DeepSeek released its paper on Engram, a novel AI architecture that incorporates conditional memory to optimize how large language models handle information. By utilizing a lookup mechanism for static patterns, this technology separates an AI's logical reasoning from its factual knowledge base. This structural shift allows massive models to run on cheaper hardware by offloading memory requirements to standard host RAM without sacrificing speed. Research indicates that this approach effectively increases model depth, freeing up the system's core processing power for more complex reasoning and long-context tasks. Ultimately, the Engram module enables superior performance across coding, math, and general logic compared to traditional architectures. This innovation suggests a future where AI is significantly more efficient and accessible through the strategic decoupling of memory and computation. Source: https://github.com/deepseek-ai/Engram/blob/main/Engram_paper.pdf

The research paper presents PageANN, a novel framework engineered to overcome the severe latency and scalability limitations facing existing disk-based Approximate Nearest Neighbor Search (ANNS) methods used in vector databases. Current systems suffer from inefficient search paths and a crucial misalignment between logical graph node size and the physical I/O granularity of Solid-State Drives (SSDs). PageANN introduces a core innovation: a page-node graph structure that directly maps logical graph nodes to physical SSD pages, significantly shortening I/O traversal paths and maximizing data utility during retrieval. This is supported by a co-designed disk data layout that embeds compressed neighbor vectors within each page and a dynamic memory management strategy utilizing lightweight indexing for fast query routing. According to experimental results, PageANN consistently outperforms state-of-the-art techniques, achieving substantial gains in throughput and latency across diverse datasets and memory constraints while maintaining comparable recall accuracy. Source: https://arxiv.org/pdf/2509.25487

This research presents a novel method for efficient long-context modeling in Large Language Models (LLMs) by tackling the quadratic complexity of attention mechanisms through KV cache compression. The core discovery is a fundamental local KV cache asymmetry, which reveals that adjacent attention keys exhibit high structural homogeneity, while their associated value vectors possess distinct, heterogeneous distributions. To capitalize on this finding, the authors propose AsymKV, a training-free compression framework that shifts information loss from heterogeneous values to homogeneous keys. AsymKV operates by applying homogeneity-based merging to keys using a mathematically derived optimal vector, paired with a lossless value representation scheme utilizing cardinality-aware normalization to preserve vital information. Extensive empirical results on benchmarks like LongBench, across diverse models such as LLaMA3.1-8B, confirm that AsymKV consistently surpasses state-of-the-art long-context methods in terms of accuracy and information retention, offering improved performance with practical inference efficiency. Source: https://arxiv.org/pdf/2506.05410

The source details the creation and evaluation of Agentic Memory (A-MEM), a novel memory system for Large Language Model (LLM) agents that addresses the fundamental rigidity of existing memory architectures. Traditional systems require predefined data structures and fixed operational workflows, which severely limits their ability to adapt to new information and maintain performance in complex, long-term tasks. A-MEM overcomes this by drawing inspiration from the Zettelkasten method, employing dynamic note construction, autonomous link generation, and memory evolution to create a self-organizing knowledge base. Experimental results on long-term dialogue datasets demonstrate that A-MEM significantly outperforms baseline methods across diverse question categories, particularly in challenging multi-hop reasoning tasks. The system is also shown to be highly efficient and scalable, requiring substantially fewer tokens for operation and maintaining minimal increases in retrieval time as the memory scale grows. These architectural advancements allow LLM agents to maintain meaningful, continuously evolving knowledge structures essential for sophisticated interaction with the environment. Source: https://openreview.net/pdf?id=FiM0M8gcct

The source introduces FlashBias, an innovative algorithm designed to significantly accelerate the efficiency of the Transformer attention mechanism when incorporating an additive bias term. Current methods, like those optimized for attention masks, cannot handle bias because these terms are generally dense and continuous rather than sparse. FlashBias overcomes this limitation by exploiting the mathematical principle that attention bias matrices exhibit an inherent low-rank structure. The technique utilizes several decomposition methods, including exact, SVD, and neural decomposition, to represent the dense bias matrix in a much smaller, compressible form. Experiments showcase substantial time and memory savings when applying FlashBias across various demanding models, such as Large Language Models, Vision Transformers, and AlphaFold 3. This new approach provides crucial efficiency for training and inference, especially for tasks involving dynamic or complex prior knowledge. Source: https://openreview.net/pdf?id=7L4NvUtZY3

The research systematically investigates the effects of integrating various gating mechanisms into the standard softmax attention layer, comparing over thirty configurations across dense and Mixture-of-Experts Large Language Models. The central finding demonstrates that applying an elementwise, head-specific sigmoid gate immediately following the Scaled Dot-Product Attention (SDPA) output consistently yields the most substantial improvement in overall performance metrics. This successful gating method also provides superior training stability, allowing models to converge effectively under larger learning rates and mitigating disruptive loss spikes during optimization. The improved efficacy is attributed to two factors: introducing essential non-linearity into the low-rank attention mapping and generating input-dependent sparse gating scores. Crucially, this sparsity acts to normalize attention dynamics, eliminating the 'attention sink' problem where initial tokens dominate attention scores, thereby facilitating notably better long-context extrapolation. These demonstrated benefits led to the incorporation of this specific gated attention design into the forthcoming Qwen3-Next models. Source: https://openreview.net/pdf?id=1b7whO4SfY

The provided text outlines DYNAACT, a new framework intended to enhance sequential reasoning in Large Language Models (LLMs) by dynamically managing the available actions during complex problem-solving. This approach targets the inefficiency of current methods that either rely on manually defined and restrictive action spaces or utilize unstructured spaces that prove computationally prohibitive for exhaustive searches. DYNAACT addresses this by first estimating a broad action space from a corpus and then using a greedy algorithm to select an optimal, compact action space for each step. The core of the method is a submodular function that ensures the selected subset of actions maintains a balance between high utility (relevance to the current state) and sufficient diversity (avoiding redundant actions). Extensive evaluation on six benchmarks confirms that DYNAACT significantly improves problem-solving accuracy—especially in math and complex reasoning tasks—while also maintaining efficient inference compared to baseline methods. Source: https://openreview.net/pdf?id=R24ZqNwoDz