AI Post Transformers

This March 2025 paper introduces compressed experts, an innovative method to enhance the efficiency of Mixture-of-Experts (MoE) models by reducing computational overhead while preserving performance. The core idea involves replacing less critical "auxiliary experts" with lightweight, compact representations, called compressed experts, during fine-tuning. This strategy allows for a significant reduction in activated parameters and inference costs—over 30% and 20% respectively, as demonstrated on models like Phi-MoE and OLMoE—while retaining more than 90% of the full model's performance. The paper details the method of identifying and aggregating these compressed experts and highlights their particular effectiveness in specialized reasoning tasks. Source: https://arxiv.org/pdf/2503.00634

This is a review of DeepSeek's latest release announced on Hugging Face on August 21, 2025. The source introduces DeepSeek-V3.1, a hybrid large language model that supports both "thinking" and "non-thinking" operational modes, distinguishable through different chat templates. This updated model offers smarter tool calling capabilities and improved thinking efficiency, providing faster responses with comparable answer quality to previous versions. Built upon a two-phase long context extension, DeepSeek-V3.1 has expanded its training dataset significantly to enhance its understanding and generation of longer documents. The document also provides detailed chat templates for various interaction types, including multi-turn conversations and tool-calling scenarios for agents, alongside evaluation metrics demonstrating its superior performance in categories like general knowledge, code, and mathematics. Finally, it outlines usage examples, local deployment instructions, and licensing information for the model. Source: https://huggingface.co/deepseek-ai/DeepSeek-V3.1

The source provides an overview of Google DeepMind's AI research and models, highlighting various applications across different scientific disciplines and creative fields. It introduces Genie 3, a general-purpose world model capable of generating diverse, interactive, real-time environments from text prompts. The document details Genie 3's capabilities, such as simulating physical properties, natural worlds, and fictional scenarios, while also addressing its limitations and the company's commitment to responsible AI development. Ultimately, the text positions Genie 3 as a significant advancement for AI research and generative media, with potential for education, training, and embodied agent development. Source: https://deepmind.google/discover/blog/genie-3-a-new-frontier-for-world-models/

We review Los Alamos National Laboratory advancements in managing indirect memory accesses in high-performance computing and it's relationship to overcoming the memory wall. The first goal of DoE’s next-generation supercomputer, ATS-5, is “Overcoming the memory wall: continued memory bandwidth performance improvements for tri-lab applications. ”"DX100" introduces a programmable data access accelerator designed to improve memory bandwidth utilization for irregular applications by reordering, coalescing, and interleaving memory requests. This accelerator aims to offload bulk indirect memory operations from CPU cores, thus reducing instruction count and cache misses. Complementing this, "A Workflow for the Synthesis of Irregular Memory Access Microbenchmarks" presents GS Patterns, a novel tool workflow that analyzes and synthesizes memory access patterns from complex applications. This workflow generates compact representations of sparse memory access patterns, enabling their use in benchmarking and hardware design to evaluate performance on various CPU and GPU architectures, particularly for gather and scatter operations that often bottleneck application performance.Sources:1) https://www.osti.gov/servlets/purl/2332770 - Codesign for memory intensive applications2) https://dl.acm.org/doi/pdf/10.1145/3695053.3731015 -DX100: Programmable Data Access Accelerator for Indirection3) https://dl.acm.org/doi/pdf/10.1145/3695794.3695816 - A Workflow for the Synthesis of Irregular Memory Access Microbenchmarks

This August 2020 paper introduces linear transformers, a novel approach to addressing the computational and memory inefficiencies of traditional transformer models, particularly for long sequences. By reframing the self-attention mechanism using a linear dot-product of kernel feature maps, the authors reduce the computational complexity from quadratic to linear, enabling significantly faster autoregressive inference. The research highlights the relationship between transformers and recurrent neural networks (RNNs), demonstrating that a causally masked transformer can be expressed as an RNN, thus allowing for constant time and memory per prediction during inference. Experimental results across image generation and speech recognition tasks show that linear transformers achieve comparable performance to standard transformers while being orders of magnitude faster and requiring less memory. Source: https://arxiv.org/pdf/2006.16236

This August 2025 survey paper explores efficient architectures for large language models (LLMs), addressing the computational challenges of models like Transformers. It categorizes advancements into linear sequence modeling, including linear attention and state-space models, which offer linear computational complexity. The document also examines sparse sequence modeling, such as static and dynamic sparse attention, designed to reduce computational demands by limiting interactions between elements. Furthermore, it discusses methods for efficient full attention, including IO-aware and grouped attention, and introduces sparse Mixture-of-Experts (MoE) models, which enhance efficiency through conditional computation. Finally, the survey highlights hybrid architectures that combine different efficient approaches and explores Diffusion LLMs and their applications across various modalities like vision and audio, underscoring the shift toward more sustainable and practical AI systems. Source: https://arxiv.org/pdf/2508.09834

This June 2022 paper introduces Switch Transformers, a novel architecture designed to enhance the efficiency and scalability of large-scale language models. Unlike traditional models that reuse the same parameters, Switch Transformers employ a Mixture-of-Experts (MoE) approach, activating different parameters for each input to achieve a sparsely-activated model with significantly more parameters at a constant computational cost. The authors simplify the MoE routing algorithm and implement improved training techniques to overcome prior limitations such as complexity, communication overhead, and instability. The paper demonstrates that Switch Transformers achieve substantial pre-training speedups and performance gains across various natural language tasks, including multilingual settings, allowing for the creation of trillion-parameter models. It also discusses the combination of data, model, and expert-parallelism for optimal scaling and the feasibility of distilling these large sparse models into smaller, more deployable dense versions. Source: https://arxiv.org/pdf/2101.03961

This April 2024 paper introduces Atom, a novel low-bit quantization method designed to enhance the efficiency and accuracy of Large Language Model (LLM) serving. The core challenge addressed is the high computational and memory costs associated with LLMs, especially when accommodating numerous user requests. Atom tackles this by quantizing both weights and activations to low-bit representations, like 4-bit, which significantly reduces memory consumption and boosts throughput by leveraging modern GPU capabilities. It maintains accuracy through mixed-precision quantization, fine-grained group quantization, and dynamic quantization, demonstrating substantial improvements in tokens per second with negligible accuracy loss compared to existing methods. The paper provides a detailed analysis of Atom's design, implementation, and comprehensive evaluation across various LLM models and tasks. Source: https://arxiv.org/pdf/2310.19102

The source analyzes Large Language Model (LLM) inference, specifically focusing on how continuous batching significantly improves efficiency compared to traditional static batching. It explains the inefficiencies of static batching where GPUs are underutilized due to varying output lengths in a batch, and introduces continuous batching (also known as dynamic batching or iteration-level scheduling) as a solution that dynamically adds new requests as others complete. The document further highlights PagedAttention and vLLM as advanced memory optimization techniques built upon continuous batching, leading to even greater throughput and reduced latency. Benchmarking results demonstrate how these innovations drastically enhance throughput and lower latency across different workloads, ultimately reducing the cost of serving LLMs. Source: https://www.anyscale.com/blog/continuous-batching-llm-inference

This August 2025 paper offers a comprehensive overview of diffusion language models (DLMs), contrasting them with traditional autoregressive (AR) and masked language models (MLMs). It highlights DLMs' unique advantages like parallel generation and iterative refinement, which address common AR model bottlenecks. The text also covers training methodologies for DLMs, including pre-training and fine-tuning techniques, as well as various inference strategies designed to enhance quality and efficiency. Furthermore, it explores the expansion of DLMs into multimodal applications and discusses current challenges and promising future research directions within the field. Source: https://arxiv.org/pdf/2508.10875