AI Post Transformers

The June 5, 2025 research paper introducing HALoS: Hierarchical Asynchronous Local SGD, a novel optimization framework designed for training large language models (LLMs) across geographically distributed accelerators and slow, high-latency networks. The core challenge addressed is the inefficiency of standard synchronous training methods due to slow inter-region communication and heterogeneous hardware speeds. HALoS mitigates these issues through a two-tier architecture featuring local parameter servers (LPSs) and a global parameter server (GPS), which leverages fast intra-region links and asynchronous updates to reduce communication overhead and minimize straggler effects. The authors provide a rigorous convergence analysis for their non-convex objective and demonstrate empirically that HALoS achieves significantly faster convergence (up to 7.5x faster than synchronous baselines) while maintaining or exceeding model quality.Sources:https://arxiv.org/pdf/2506.04531

The June 7, 2025 UT Austin and University of British Colombia collaboration academic paper introduces MorphKV, a novel inference-time technique designed to address the excessive memory consumption caused by Key-Value (KV) caches in Large Language Models (LLMs) during extended responses. The core problem is that KV cache size grows linearly with sequence length, straining GPU memory, leading to prior methods sacrificing accuracy by dropping context or using lossy compression. MorphKV resolves this by maintaining a constant-sized KV cache through a dynamic, correlation-aware token selection mechanism that retains the most relevant older tokens based on the attention profiles of recent tokens. Evaluations on long-response tasks, such as content creation and code generation, demonstrate that MorphKV achieves significant memory savings (up to 52.9%) while delivering higher accuracy (up to 18.2%) compared to state-of-the-art compression methods like SnapKV and H2O. The research emphasizes the distinction between long-context and long-response tasks, positioning MorphKV as a robust solution particularly for the latter by efficiently managing memory throughout the decoding phase. Source: https://arxiv.org/pdf/2503.00979

The October 9, 2025 paper from UT Austin paper introduces PolicySmith, a novel framework that automates the design of system policies, arguing that the traditional manual creation of heuristics by experts is becoming inefficient due to rapidly changing environments. PolicySmith leverages Large Language Models (LLMs) and evolutionary search to generate instance-optimal heuristic code that is tailored to specific workloads and hardware contexts. The authors demonstrate the framework's effectiveness in two critical systems domains: discovering superior cache eviction policies for web caching and generating functional, safe policies for Linux kernel congestion control through eBPF. This research proposes a fundamental shift, moving policy intelligence from fixed rules to an automated process of code generation, which results in more performant and context-aware system policies compared to established human-designed and pure machine-learning baselines. Source: https://arxiv.org/pdf/2510.08803

The October 28, 2025 Samsung research paper introduces zFLoRA (zero-latency fused low-rank adapter), a novel parameter-efficient fine-tuning (PEFT) method designed to address the significant inference latency overheads associated with current adapter methods like LoRA in large language models (LLMs). The core contribution is a carefully engineered fusion of adapter blocks with the base model to achieve zero or negligible latency overhead during inference, leveraging optimized matrix multiplication on hardware like NVIDIA H100 GPU and Samsung Galaxy S25+ NPU. Experimental results across LLMs ranging from 1B to 7B parameters demonstrate that zFLoRA maintains performance comparable to LoRA and Full Fine-Tuning (FFT) across reasoning and generation tasks, while effectively eliminating the latency penalty, as visually confirmed by accompanying bar graphs. The paper details the architectural design of zFLoRA, which avoids costly expansion and merge operations present in naive fused adapter designs, and includes extensive latency measurements validating its efficiency on various platforms. Source: https://arxiv.org/pdf/2510.25784

The August 26, 2025 collaboration between the University of Washington, NVIDIA and the Allen Institute for AI paper introduces "SuperBPE: Space Travel for Language Models," introduces SuperBPE, a novel tokenization method that challenges the standard practice of limiting tokens to subword boundaries. The authors argue that conventional Byte-Pair Encoding (BPE) is inefficient because it cannot create "superword" tokens that bridge whitespace, ignoring common multi-word expressions that function as single semantic units. SuperBPE addresses this by incorporating a two-stage curriculum into BPE, first learning subwords and then learning superwords, resulting in up to 33% fewer tokens needed to encode text. Experiments with 8B transformer Language Models (LMs) demonstrate that models trained with SuperBPE achieve an average improvement of +4.0% across 30 downstream tasks and require 27% less compute at inference time compared to BPE baselines. The analysis suggests SuperBPE's success stems from creating more uniform per-token difficulty by capturing these cohesive multi-word expressions. Source: https://arxiv.org/pdf/2503.13423

The October 30, 2025 technical report details the development and evaluation of Kimi Linear, a novel hybrid linear attention architecture for large language models (LLMs). The core innovation is the Kimi Delta Attention (KDA) module, which refines existing linear attention mechanisms to achieve superior performance and efficiency compared to traditional full attention, particularly in long-context scenarios. Empirical results from extensive pretraining and fine-tuning experiments demonstrate that Kimi Linear outperforms baselines across various tasks, including general reasoning and code generation, while significantly reducing memory usage and increasing decoding throughput. The report also includes a complexity analysis and a detailed discussion of KDA's relationship to other efficient attention and state-space models. Source: https://arxiv.org/pdf/2510.26692

The August 9, 2023 paper introduces the Retentive Network (RetNet), a proposed foundational architecture for large language models intended to succeed the Transformer model. RetNet aims to overcome the Transformer's inefficiencies during inference by simultaneously achieving training parallelism, low-cost inference, and strong performance, a combination previously considered an "impossible triangle." The core of RetNet is the retention mechanism, which supports three computation paradigms—parallel, recurrent, and chunkwise recurrent—to enable efficient training and constant-time, O(1) inference, leading to significant reductions in GPU memory, latency, and increased throughput compared to the Transformer. Experimental results across various model sizes and tasks demonstrate that RetNet is competitive in performance and offers superior efficiency in both training and deployment. Source: https://arxiv.org/pdf/2307.08621

The April 22, 2022 collaboration between University of Washington, Facebook AI and the Allen Institute for AI introduces Attention with Linear Biases (ALiBi), a novel and efficient method for position representation in transformer models that effectively addresses the challenge of extrapolation—a model's ability to maintain performance on input sequences longer than those used during training. The authors demonstrate that traditional position encoding methods, like sinusoidal embeddings, fail to extrapolate efficiently, while alternatives like the T5 bias are computationally costly. ALiBi improves extrapolation by biasing query-key attention scores with a distance-proportional penalty, eliminating the need for positional embeddings entirely. This approach is shown to be faster and more memory-efficient than baselines, enabling a large 1.3 billion parameter model trained on shorter sequences to achieve comparable or superior perplexity scores when evaluated on significantly longer sequences. The findings suggest that ALiBi's performance gains when extrapolating are primarily due to mitigating the "early token curse" common in sequence-splitting evaluation methods.

On October 29, 2025 Anthropic presented research investigating the existence of functional introspective awareness in large language models (LLMs), specifically focusing on Anthropic's Claude models. The core methodology involves using concept injection, where researchers manipulate a model's internal activations with representations of specific concepts to see if the model can accurately report on these altered internal states. Experiments demonstrate that models can, at times, notice injected "thoughts," distinguish these internal representations from text inputs, detect when pre-filled outputs were unintentional by referring to prior intentions, and even modulate their internal states when instructed to "think about" a concept. The findings indicate that while this introspective capacity is often unreliable and context-dependent, the most capable models, such as Claude Opus 4 and 4.1, exhibit the strongest signs of this ability, suggesting it may emerge with increased model sophistication. Source: https://transformer-circuits.pub/2025/introspection/index.html

The October 21, 2025 collaboration paper between UW-Madison and Amazon Web Services discuss the critical role of the Multi-Layer Perceptron (MLP) intermediate size f_size as the primary architectural component for introducing non-linearity and complexity within Large Language Models (LLMs). The MLP layer achieves this by taking the hidden state d_model projecting it up to the expanded f_size, applying a non-linear gating function (like SwiGLU), and then projecting it back down. The balance between the MLP and the attention layers is governed by the mlp-to-attention ratio r_mlp/attn, which is essential for maximizing accuracy (by minimizing training loss) and optimizing inference efficiency (by boosting throughput). Extensive scaling law analysis demonstrates that both the hidden size and the r_mlp/attn exhibit a U-shaped relationship with training loss, confirming that careful tuning of these architectural parameters is necessary to achieve optimal model performance and inference speed. Source: https://arxiv.org/pdf/2510.18245