The October 21, 2025 collaboration paper between UW-Madison and Amazon Web Services discuss the critical role of the **Multi-Layer Perceptron (MLP) intermediate size f_size as the primary architectural component for introducing non-linearity and complexity within Large Language Models (LLMs). The MLP layer achieves this by taking the hidden state d_model projecting it up to the expanded f_size, applying a **non-linear gating function** (like SwiGLU), and then projecting it back down. The balance between the MLP and the attention layers is governed by the **mlp-to-attention ratio r_mlp/attn, which is essential for maximizing accuracy (by minimizing training loss) and optimizing inference efficiency (by boosting throughput). Extensive scaling law analysis demonstrates that both the hidden size and the r_mlp/attn exhibit a **U-shaped relationship with training loss**, confirming that careful tuning of these architectural parameters is necessary to achieve optimal model performance and inference speed.

Source:

https://arxiv.org/pdf/2510.18245

We cover two new innovations from Microsoft extending ideas from the original old **FlashAttention**. Flash Attention is an IO-aware attention algorithm for Transformers designed to address the quadratic time and memory complexity of standard self-attention on long sequences. By using **tiling and recomputation** to minimize slow **High Bandwidth Memory (HBM)** accesses in favor of fast **on-chip SRAM**, FlashAttention achieves significant wall-clock speedups for training models like BERT and GPT-2, enabling them to handle much longer context lengths. Microsoft's new **ATTENTION2D** is a technique that builds upon memory-efficient methods like FlashAttention to optimize **distributed self-attention** across multiple GPUs, achieving parallelism in two dimensions (Q-DIM and KV-DIM) to overcome the communication bottleneck inherent in prior single-dimension parallel approaches like Ring Attention. Microsoft's additional contribution to the research community is **Lean Attention**, which also appears to propose a high-performance, tiled execution strategy for attention, using shared memory and iterative computation, similar to the IO-aware concepts in the other sources.

Sources:

The original flag attention paper:

https://arxiv.org/pdf/2205.14135

Flash attention 2 paper:

https://arxiv.org/pdf/2307.08691

June 28, 2025 Microsoft's Attention2D:

https://arxiv.org/pdf/2503.15758

Microsoft's Lean attention:

https://www.microsoft.com/en-us/research/wp-content/uploads/2024/05/Lean_Attention___arxiv_version.pdf

The provided text introduces **Sentence-BERT (SBERT)**, a modification of the popular **BERT** and **RoBERTa** language models, designed to efficiently generate **semantically meaningful sentence embeddings**. The authors address the significant **computational overhead** of using standard BERT for tasks requiring sentence-pair comparisons, such as semantic similarity search and clustering, which can take hours for large datasets. SBERT utilizes **siamese and triplet network structures** to create fixed-size sentence vectors that can be quickly compared using metrics like **cosine-similarity**, drastically reducing the computation time from hours to seconds while **maintaining or exceeding accuracy**. Evaluation results demonstrate that SBERT significantly **outperforms other state-of-the-art sentence embedding methods** on various Semantic Textual Similarity (STS) and transfer learning tasks. Ultimately, SBERT makes **BERT usable for large-scale applications** where the original architecture was too slow.

Source:

https://arxiv.org/pdf/1908.10084

The source provides excerpts from a scientific paper introducing **TxGNN**, a novel graph foundation model designed for **zero-shot drug repurposing**, which aims to identify therapeutic candidates even for diseases with no existing treatments or limited molecular data. Developed by researchers affiliated with institutions like Harvard Medical School and Stanford University, this model leverages a **medical knowledge graph (KG)** and a graph neural network (GNN) to predict drug indications and contraindications across over 17,000 diseases, demonstrating significant performance improvements over existing methods. The paper highlights TxGNN’s ability to generate **multi-hop interpretable explanations** for its predictions, fostering trust and aiding human experts, and validates its clinical relevance by showing alignment with **off-label prescriptions** observed in electronic medical records (EMRs). Overall, the work presents a comprehensive AI framework to systemize and enhance drug repurposing, particularly for neglected or rare diseases.

Source:

https://pmc.ncbi.nlm.nih.gov/articles/PMC11645266/

These April 29, 2024 paper provides an overview of the challenges associated with using **NVIDIA's Multi-Instance GPU (MIG)** technology, specifically focusing on the address translation mechanism in the **A100 GPU**. The papers reveal, primarily through **reverse-engineering efforts**, that the L2 and L3 Translation Lookaside Buffers (**TLBs**) utilize a compression design where each entry comprises **16 sub-entries** to enhance memory capacity management. A major problem arises because the **L3 TLB is shared** across all isolated MIG instances, causing contention that results in frequent evictions and low utilization of these sub-entries. To mitigate this performance degradation, the sources propose **STAR**, a novel hardware solution that dynamically enables the sharing of TLB sub-entries among different base addresses to improve overall efficiency.

Source:

https://arxiv.org/pdf/2404.18361

The August 26, 2025 collaboration between Stanford, NVIDIA, Shanghai Jiao Tong University, University of Michigan, University of Colorado Boulder, Carnegie Mellon University introduces **Strata**, a hierarchical context caching framework designed to improve the performance of serving Large Language Models (LLMs) with long context windows. The core problem Strata addresses is that while caching key-value (KV) states is essential for efficiency, transferring large, fragmented cached contexts from slower memory tiers (like CPU memory) back to the GPU creates **severe I/O bottlenecks and performance stalls**. It also describes why paged attention creates data fragmentation when offloading even though its goal is to address memory fragmentation. That is paged attention becomes an issue when using offloading due to large contexts. Strata overcomes these issues through two main innovations: **GPU-assisted I/O** to mitigate data fragmentation and achieve high bandwidth utilization, and **cache-aware request scheduling** to intelligently form balanced batches and overlap unavoidable I/O stalls with complementary tasks. The evaluation shows that Strata significantly reduces the **Time-To-First-Token (TTFT)** and increases throughput compared to state-of-the-art serving systems like vLLM + LMCache and TensorRT-LLM on long-context benchmarks.

Source:

https://arxiv.org/html/2508.18572v1

This is a classic review of a now old but yet still important paper, the original Flash Attention paper. We review this in light of advances in compiler technology.

The June 23, 2022 Stanford paper describes the original **FlashAttention**, an innovative, IO-aware algorithm designed to significantly enhance the efficiency of the attention mechanism in Transformer models by optimizing memory usage and access. Standard attention suffers from complexity that scales **quadratically** ($O(N^2)$) with sequence length ($N$) for both memory footprint and access to slow High Bandwidth Memory (HBM), which creates a performance bottleneck. FlashAttention overcomes this by employing **tiling and recomputation** within a single customized CUDA kernel, dramatically reducing the memory footprint to scale **linearly** ($O(N)$) and eliminating the quadratic term in HBM access complexity. While the algorithm does not reduce the total Floating Point Operations (FLOPs) and even slightly increases them due to recomputation, the massive reduction in slow memory transfers results in substantial **wall-clock runtime speedups** during both training and inference.

Source:

https://arxiv.org/pdf/2205.14135

On October 20, 2025 Hugging Face released **MTEB v2**, a significant refactoring of the Massive Text Embedding Benchmark, which was originally designed for evaluating text embedding models across various tasks like classification and retrieval. The update addresses **package bloating and the need for broader support** by introducing a **more consistent interface, better typing, and improved documentation**. Key new features include support for **multimodal evaluation (text, images, and audio)**, **unified retrieval and reranking tasks**, and an **easier evaluation process** using the new `mteb.evaluate` function and `ResultCache` for managing results. The article also provides detailed instructions for **upgrading from MTEB v1**, including how to convert old models and datasets to the new v2 format.

Source:

https://huggingface.co/blog/isaacchung/mteb-v2

The October 10, 2025 paper from the University of Michigan and **Google DeepMind** concerning the phenomenon of **"overthinking" in Large Language Models (LLMs)** that utilize chain-of-thought (**CoT**) reasoning. The authors introduce a systematic analyzer called **TRACE** to structurally examine an LLM's thought process, decomposing it into sub-thoughts and progression graphs to move beyond superficial, length-based metrics of overthinking. Benchmarking across various tasks reveals that "thinking models" often waste significant computational resources on simple queries without notable accuracy gains, operating **five to twenty times slower** than non-thinking counterparts. The study identifies two primary overthinking patterns—**Explorer** (characterized by over-exploration and backtracking) and **Late Landing** (marked by excessive self-verification)—and proposes a **utility-based redefinition of overthinking** focused on diminishing marginal returns of subsequent thoughts.

Source:

https://arxiv.org/pdf/2510.07880

The October 23 2025 research paper **probes the spatial reasoning capabilities of Large Language Models (LLMs) when processing text-based inputs**, specifically focusing on how performance degrades as task complexity increases. Using a suite of five grid-based tasks—including quadrant identification, geometric transformations, distance evaluation, word searches, and tile sliding—the authors tested four models: GPT-4o, GPT-4.1, and two variants of Claude 3.7. The key finding is that while models achieve **moderate success on smaller grids**, their accuracy rapidly deteriorates as grid dimensions scale up, demonstrating a **significant gap between linguistic and robust spatial representation** in their architectures. Notably, the **Anthropic models consistently outperformed the OpenAI variants**, though all models exhibited weaknesses, such as frequent miscounting, mathematical errors, and difficulty maintaining board state in complex scenarios. The study concludes by emphasizing the **fragility of LLM spatial reasoning** at scale and suggesting future work on improving text-based spatial data representation and mathematical capabilities.

Source:

https://arxiv.org/pdf/2510.20198