AI Post Transformers

The provided text introduces DeepSeekMoE, an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in large language models. The authors propose two key strategies: fine-grained expert segmentation, which divides experts into smaller, more numerous units for flexible combinations, and shared expert isolation, which designates specific experts for common knowledge to reduce redundancy. Through comprehensive experimentation, DeepSeekMoE demonstrates superior performance and computational efficiency compared to conventional MoE models like GShard and dense models, even when scaled up to 145B parameters. The research also highlights DeepSeekMoE's adaptability for fine-tuning into chat models and emphasizes its lower redundancy among routed experts, ultimately aiming for more accurate and efficient knowledge acquisition. Source: 2024 - https://arxiv.org/pdf/2401.06066

This paper introduces DeepSeek-V3, a large Mixture-of-Experts (MoE) model designed to advance open-source language model capabilities with improved training efficiency and performance. The document details its innovative architecture, including an auxiliary-loss-free load balancing strategy and a Multi-Token Prediction objective for enhanced data efficiency and future token prediction. It further explains the infrastructures and optimizations that enable its cost-effective training, such as efficient communication protocols and a low-precision training framework using FP8. Finally, the paper outlines DeepSeek-V3's pre-training and post-training processes, including its long context extension capabilities and knowledge distillation techniques from the DeepSeek-R1 series, along with comprehensive evaluations across various benchmarks demonstrating its strong performance, especially in coding and mathematics. Source: https://arxiv.org/pdf/2412.19437

This paper introduces DeepSeek-R1, a new suite of large language models developed by DeepSeek-AI, focusing on enhancing reasoning capabilities through reinforcement learning (RL). It details the development of DeepSeek-R1-Zero, a model trained purely with RL that demonstrates strong reasoning but has readability issues, and DeepSeek-R1, which addresses these flaws by incorporating multi-stage training with initial "cold-start" data and achieves performance comparable to OpenAI-o1-1217. The document also covers the distillation of reasoning abilities from larger DeepSeek-R1 models into smaller, more efficient models, making them available to the research community. Performance benchmarks on various tasks, including mathematics, coding, and general knowledge, are presented, highlighting the models' advancements. The paper concludes by discussing the effectiveness of distillation versus direct RL on smaller models and outlines future research directions. Source: https://arxiv.org/pdf/2501.12948

This document explores the Mamba architecture, a novel approach to sequence modeling that offers an efficient alternative to Transformers. It primarily investigates the role of "input selectivity" within Mamba's core component, the S6 layer, and its impact on the model's capabilities. The research proves Mamba's superiority over its predecessor, S4D, in approximating discontinuous functions and demonstrates how input selectivity helps counteract memory decay for long sequences. Furthermore, the paper analyzes how the complete Mamba architecture, including convolution and gating, efficiently solves complex associative recall tasks like Multiple-Query Associative Recall (MQAR) and Induction Heads, with theoretical bounds on model size confirmed by empirical results. The findings offer a mechanistic understanding of Mamba's performance and suggest pathways for future enhancements, such as optimizing input dependence within its state matrix. Source: https://arxiv.org/pdf/2506.11891

This paper introduces a novel deep unsupervised learning algorithm that leverages non-equilibrium thermodynamics to model complex datasets. The core idea involves a forward diffusion process that systematically degrades data structure, and then learning a reverse diffusion process to reconstruct it, creating a flexible generative model. This approach allows for efficient learning, sampling, and probability evaluation in deep generative models, even with numerous layers. The authors demonstrate the method's efficacy on various datasets, including images, highlighting its ability to handle tasks like denoising and inpainting by multiplying distributions.

This research paper focuses on a safety evaluation of DeepSeek-R1 and DeepSeek-V3 models within Chinese language contexts, an area previously underexplored. It highlights that while DeepSeek models possess strong reasoning capabilities, previous studies, primarily in English, have revealed significant safety flaws. To address the gap in Chinese safety assessments, the authors introduce CHiSafetyBench, a new benchmark designed to systematically test these models across various safety categories like discrimination and violation of values. The experimental results quantitatively demonstrate the deficiencies of DeepSeek models in Chinese safety performance, particularly in identifying and refusing harmful content, offering insights for future improvements. The authors acknowledge potential biases in their evaluation and plan to continually optimize the benchmark. Source: https://arxiv.org/pdf/2502.11137

This academic paper introduces DiMSUM, a novel architecture for image generation that enhances diffusion models by integrating both spatial and frequency information. The authors address limitations of existing state-space models like Mamba in handling image data by incorporating wavelet transformations and a cross-attention fusion layer, which better captures both local details and long-range dependencies. Furthermore, the model includes globally shared transformer blocks to improve global relationship modeling, a known weakness of Mamba. Experiments show that DiMSUM achieves superior image quality and faster training convergence compared to current state-of-the-art models on various benchmarks. Source: 2025 - https://arxiv.org/pdf/2411.04168 - DiMSUM : Diffusion Mamba - A Scalable and Unified Spatial-Frequency Meth

The provided text introduces DroidSpeak, a novel distributed Large Language Model (LLM) inference system designed to enhance the efficiency of compound AI systems. It addresses the challenge of reusing Key-Value (KV) caches across different LLMs that share the same architectural foundation, a problem current systems struggle with. DroidSpeak achieves significant throughput improvements and faster prefill times by selectively recomputing only a small, "critical" subset of KV cache layers, while reusing the rest, with negligible impact on quality. This selective recomputation, determined through an offline profiling stage, is further optimized by pipelining re-computation with KV cache loading, making it practical for multi-LLM workflows in distributed settings. The paper demonstrates DroidSpeak's robustness and benefits across various tasks and model pairs. Source: Published July 2025 https://arxiv.org/pdf/2411.02820v4

The paper introduces Dynamic Tanh (DyT), a novel element-wise operation designed to replace normalization layers in Transformer models. Traditionally, normalization layers like Layer Normalization (LN) are considered essential for stable and effective training of deep neural networks. However, this paper challenges that belief by demonstrating that DyT, which mimics the S-shaped input-output mapping observed in LN, can achieve comparable or superior performance across various tasks, including vision, language, and speech processing, often without extensive hyperparameter tuning. This research suggests that the non-linear squashing of extreme values by normalization layers, rather than their statistical normalization, is a key mechanism, offering new insights into their role in deep learning architectures. While effective for Transformers, preliminary findings suggest DyT may not directly substitute Batch Normalization (BN) in classic ConvNets. Source: Published June 2025 https://arxiv.org/pdf/2503.10622

This document introduces DyNN-Offload, a novel memory management system designed to overcome the GPU memory limitations faced when training large dynamic neural networks (DyNNs). Unlike traditional methods that struggle with DyNNs' unpredictable memory access patterns, DyNN-Offload employs a learned approach using a lightweight "pilot model" to predict tensor access orders. By using an idiom-based representation of network operations, the pilot model efficiently guides the migration of tensors between CPU and GPU memory, enabling significantly larger DyNN training on a single GPU. The system demonstrates superior performance compared to existing solutions like unified virtual memory (UVM) and dynamic tensor rematerialization (DTR), while introducing minimal overhead. Its transparent integration with existing deep learning frameworks makes it a practical solution for advancing large-scale DyNN development. Source: 2024 - https://web.cs.ucla.edu/~harryxu/papers/ren-hpca24.pdf - Enabling Large Dynamic Neural Network Training with Learning-based Memory Management