This June 2022 paper introduces Switch Transformers, a novel architecture designed to enhance the efficiency and scalability of large-scale language models. Unlike traditional models that reuse the same parameters, Switch Transformers employ a Mixture-of-Experts (MoE) approach, activating different parameters for each input to achieve a sparsely-activated model with significantly more parameters at a constant computational cost. The authors simplify the MoE routing algorithm and implement improved training techniques to overcome prior limitations such as complexity, communication overhead, and instability. The paper demonstrates that Switch Transformers achieve substantial pre-training speedups and performance gains across various natural language tasks, including multilingual settings, allowing for the creation of trillion-parameter models. It also discusses the combination of data, model, and expert-parallelism for optimal scaling and the feasibility of distilling these large sparse models into smaller, more deployable dense versions.

Source: https://arxiv.org/pdf/2101.03961

This August 2020 paper introduces linear transformers, a novel approach to addressing the computational and memory inefficiencies of traditional transformer models, particularly for long sequences. By reframing the self-attention mechanism using a linear dot-product of kernel feature maps, the authors reduce the computational complexity from quadratic to linear, enabling significantly faster autoregressive inference. The research highlights the relationship between transformers and recurrent neural networks (RNNs), demonstrating that a causally masked transformer can be expressed as an RNN, thus allowing for constant time and memory per prediction during inference. Experimental results across image generation and speech recognition tasks show that linear transformers achieve comparable performance to standard transformers while being orders of magnitude faster and requiring less memory.

Source: https://arxiv.org/pdf/2006.16236

This August 2025 survey paper explores efficient architectures for large language models (LLMs), addressing the computational challenges of models like Transformers. It categorizes advancements into linear sequence modeling, including linear attention and state-space models, which offer linear computational complexity. The document also examines sparse sequence modeling, such as static and dynamic sparse attention, designed to reduce computational demands by limiting interactions between elements. Furthermore, it discusses methods for efficient full attention, including IO-aware and grouped attention, and introduces sparse Mixture-of-Experts (MoE) models, which enhance efficiency through conditional computation. Finally, the survey highlights hybrid architectures that combine different efficient approaches and explores Diffusion LLMs and their applications across various modalities like vision and audio, underscoring the shift toward more sustainable and practical AI systems.

Source:

https://arxiv.org/pdf/2508.09834

This April 2024 paper introduces Atom, a novel low-bit quantization method designed to enhance the efficiency and accuracy of Large Language Model (LLM) serving. The core challenge addressed is the high computational and memory costs associated with LLMs, especially when accommodating numerous user requests. Atom tackles this by quantizing both weights and activations to low-bit representations, like 4-bit, which significantly reduces memory consumption and boosts throughput by leveraging modern GPU capabilities. It maintains accuracy through mixed-precision quantization, fine-grained group quantization, and dynamic quantization, demonstrating substantial improvements in tokens per second with negligible accuracy loss compared to existing methods. The paper provides a detailed analysis of Atom's design, implementation, and comprehensive evaluation across various LLM models and tasks.

Source: https://arxiv.org/pdf/2310.19102

The source analyzes Large Language Model (LLM) inference, specifically focusing on how continuous batching significantly improves efficiency compared to traditional static batching. It explains the inefficiencies of static batching where GPUs are underutilized due to varying output lengths in a batch, and introduces continuous batching (also known as dynamic batching or iteration-level scheduling) as a solution that dynamically adds new requests as others complete. The document further highlights PagedAttention and vLLM as advanced memory optimization techniques built upon continuous batching, leading to even greater throughput and reduced latency. Benchmarking results demonstrate how these innovations drastically enhance throughput and lower latency across different workloads, ultimately reducing the cost of serving LLMs.

Source: https://www.anyscale.com/blog/continuous-batching-llm-inference

This August 2025 paper introduces Self-Search Reinforcement Learning (SSRL), a novel method that enables Large Language Models (LLMs) to access and utilize their internal knowledge for search-driven tasks, bypassing the need for external search engines like Google or Bing. The research explores how repeated sampling can enhance an LLM's intrinsic search capabilities and investigates the impact of various prompting strategies and training methodologies, including the benefits of information masking and format-based rewards. The paper demonstrates that SSRL-trained models can effectively generalize to real-world search scenarios while often outperforming methods that rely on external search APIs, suggesting LLMs can function as powerful internal knowledge bases for complex queries.

Source:

https://arxiv.org/pdf/2508.10874

This August 2025 paper offers a comprehensive overview of diffusion language models (DLMs), contrasting them with traditional autoregressive (AR) and masked language models (MLMs). It highlights DLMs' unique advantages like parallel generation and iterative refinement, which address common AR model bottlenecks. The text also covers training methodologies for DLMs, including pre-training and fine-tuning techniques, as well as various inference strategies designed to enhance quality and efficiency. Furthermore, it explores the expansion of DLMs into multimodal applications and discusses current challenges and promising future research directions within the field.

Source:

https://arxiv.org/pdf/2508.10875

This 2022 paper is a reminder of issues with mmap() for databases. Yet many Vector Databases today rely on mmap().

This academic paper critically evaluates the use of memory-mapped file I/O (mmap) in Database Management Systems (DBMSs), arguing against its perceived benefits over traditional buffer pool implementations. The authors explain that while mmap appears to simplify file I/O by letting the Operating System (OS) handle data movement between storage and memory, it introduces significant correctness and performance issues. They detail problems concerning transactional safety, I/O stalls, error handling, and performance bottlenecks like TLB shootdowns, illustrating these points with experimental analysis. The paper concludes by advising against mmap for most DBMS applications, especially those requiring high throughput or transactional safety.

Source: https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf

This is a huge review of 13 different sources on advancements in GPU-accelerated computing, focusing on data access, memory management, and performance optimization for large datasets. Several sources highlight NVIDIA's initiatives like GPUDirect Storage and the AI Data Platform, which streamline data transfer directly between storage and GPUs, reducing CPU bottlenecks. Conversely, other documents analyze AMD's efforts with ROCm, acknowledging its rapid software stack improvements but also pointing out challenges like lack of comprehensive Python support and the need for increased R&D investment to compete with NVIDIA's established CUDA ecosystem. Concepts such as GPU-orchestrated memory tiering and novel I/O primitives are presented as solutions to overcome limitations in GPU memory capacity and PCIe bandwidth, enabling more efficient processing of extensive data analytics and AI workloads.

Source 1: GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture

https://arxiv.org/pdf/2203.04910

Source 2: Vortex: Overcoming Memory Capacity Limitations in GPU-Accelerated Large-Scale Data Analytics

https://arxiv.org/pdf/2502.09541

Source 3: GPU as Data Access Engines

https://files.futurememorystorage.com/proceedings/2024/20240808_NETC-301-1_Newburn.pdf

Source 4: Performance Analysis of Different IO Methods between GPU Memory and Storage

https://www.tkl.iis.u-tokyo.ac.jp/new/uploads/publication_file/file/1051/6C-03.pdf

Source 5: GDS cuFile API Reference - https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

Source 6: AMD 2.0 – New Sense of Urgency | MI450X Chance to Beat Nvidia | Nvidia’s New Moat Rapid Improvements, Developers First Approach, Low AMD AI Software Engineer Pay, Python DSL, UALink Disaster, MI325x, MI355x, MI430X UL4, MI450X Architecture, IF64/IF128, Flexible IO, UALink, IFoE https://semianalysis.com/2025/04/23/amd-2-0-new-sense-of-urgency-mi450x-chance-to-beat-nvidia-nvidias-new-moat/

Source 7: Accelerating and Securing GPU Accesses to Large Datasets

https://www.nvidia.com/en-us/on-demand/session/gtc24-s62559/

Source 8: GMT: GPU Orchestrated Memory Tiering for the Big Data Era

https://dl.acm.org/doi/10.1145/3620666.3651353

Source 9: GPUDirect Storage

https://docs.nvidia.com/gpudirect-storage/

Source 10: GPUDirect Storage: A Direct Path Between Storage and GPU Memory

https://developer.nvidia.com/blog/gpudirect-storage/

Source 11: Introducing ROCm-DS: GPU-Accelerated Data Science for AMD Instinct™ GPUs

https://rocm.blogs.amd.com/software-tools-optimization/introducing-rocm-ds-revolutionizing-data-processing-with-amd-instinct-gpus/README.html

Source 12: NVIDIA and Storage Industry Leaders Unveil New Class of Enterprise Infrastructure for the Age of AI

https://nvidianews.nvidia.com/news/nvidia-and-storage-industry-leaders-unveil-new-class-of-enterprise-infrastructure-for-the-age-of-ai

Source 13: Why is CUDA so much faster than ROCm?

https://www.reddit.com/r/MachineLearning/comments/1fa8vq5/d_why_is_cuda_so_much_faster_than_rocm/

This reviews the IETF Parallel Network File System (pNFS), an extension to NFS that separates file metadata from data storage. Specifically, "RFC 8435" introduces the initial flexible file layout type, enabling pNFS to utilize existing protocols and storage devices with limited metadata server interaction, incorporating client-side mirroring for data replication. Then we review, "Parallel NFS (pNFS) Flexible File Layout Version 2," details an update to this layout, enhancing data protection with additional methods like the Mojette algorithm, and refining the management of tightly and loosely coupled storage device models, including how client I/O errors and layout usage statistics are reported and managed. Both sources discuss the mechanisms for recalling layouts, client fencing, and the security considerations inherent in these distributed file system architectures.

Sources: https://www.ietf.org/archive/id/draft-ietf-nfsv4-layoutwcc-02.html - Add LAYOUT_WCC to NFSv4.2's Flex File Layout Type

https://www.ietf.org/archive/id/draft-haynes-nfsv4-flexfiles-v2-00.html - Parallel NFS (pNFS) Flexible File Layout Version 2

https://datatracker.ietf.org/doc/rfc8435/ - Parallel NFS (pNFS) Flexible File Layout

RFC 8435