Sign up to save your podcasts
Or
Share SonicMoE- Accelerating MoE with IO and Tile-aware Optimizations
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
SonicMoE- Accelerating MoE with IO and Tile-aware Optimizations
In this episode:
• Introduction to SonicMoE: Professor Norris and Linda introduce the episode's topic, the SonicMoE paper, and discuss the recent trends toward fine-grained and highly sparse Mixture of Experts models.
• The Hardware Inefficiency Problem: The hosts break down why increasing MoE granularity and sparsity leads to major hardware bottlenecks, specifically focusing on IO costs, activation memory, and tile quantization effects.
• Minimizing Activation Memory: Linda explains SonicMoE's clever algorithmic redesign of the backward pass computation graph, which reduces activation memory by 45 percent without adding any extra FLOPs.
• Overlapping Compute and IO: The discussion shifts to GPU-level optimizations. Linda details how the authors use Ping-Pong scheduling and asynchronous TMA on Hopper GPUs to hide memory latency behind matrix math.
• Token Rounding Routing: Professor Norris questions the padding waste in Grouped GEMMs. Linda reveals the paper's novel Token Rounding method that aligns token routing exactly with hardware tile sizes, saving massive compute.
• Conclusion and Impact: The hosts wrap up the episode by discussing the broader implications of SonicMoE for the future of large language model training and scaling laws.
• Introduction to SonicMoE: Professor Norris and Linda introduce the episode's topic, the SonicMoE paper, and discuss the recent trends toward fine-grained and highly sparse Mixture of Experts models.
• The Hardware Inefficiency Problem: The hosts break down why increasing MoE granularity and sparsity leads to major hardware bottlenecks, specifically focusing on IO costs, activation memory, and tile quantization effects.
• Minimizing Activation Memory: Linda explains SonicMoE's clever algorithmic redesign of the backward pass computation graph, which reduces activation memory by 45 percent without adding any extra FLOPs.
• Overlapping Compute and IO: The discussion shifts to GPU-level optimizations. Linda details how the authors use Ping-Pong scheduling and asynchronous TMA on Hopper GPUs to hide memory latency behind matrix math.
• Token Rounding Routing: Professor Norris questions the padding waste in Grouped GEMMs. Linda reveals the paper's novel Token Rounding method that aligns token routing exactly with hardware tile sizes, saving massive compute.
• Conclusion and Impact: The hosts wrap up the episode by discussing the broader implications of SonicMoE for the future of large language model training and scaling laws.
View all episodes
By Mechanical DirkIn this episode:
• Introduction to SonicMoE: Professor Norris and Linda introduce the episode's topic, the SonicMoE paper, and discuss the recent trends toward fine-grained and highly sparse Mixture of Experts models.
• The Hardware Inefficiency Problem: The hosts break down why increasing MoE granularity and sparsity leads to major hardware bottlenecks, specifically focusing on IO costs, activation memory, and tile quantization effects.
• Minimizing Activation Memory: Linda explains SonicMoE's clever algorithmic redesign of the backward pass computation graph, which reduces activation memory by 45 percent without adding any extra FLOPs.
• Overlapping Compute and IO: The discussion shifts to GPU-level optimizations. Linda details how the authors use Ping-Pong scheduling and asynchronous TMA on Hopper GPUs to hide memory latency behind matrix math.
• Token Rounding Routing: Professor Norris questions the padding waste in Grouped GEMMs. Linda reveals the paper's novel Token Rounding method that aligns token routing exactly with hardware tile sizes, saving massive compute.
• Conclusion and Impact: The hosts wrap up the episode by discussing the broader implications of SonicMoE for the future of large language model training and scaling laws.
• Introduction to SonicMoE: Professor Norris and Linda introduce the episode's topic, the SonicMoE paper, and discuss the recent trends toward fine-grained and highly sparse Mixture of Experts models.
• The Hardware Inefficiency Problem: The hosts break down why increasing MoE granularity and sparsity leads to major hardware bottlenecks, specifically focusing on IO costs, activation memory, and tile quantization effects.
• Minimizing Activation Memory: Linda explains SonicMoE's clever algorithmic redesign of the backward pass computation graph, which reduces activation memory by 45 percent without adding any extra FLOPs.
• Overlapping Compute and IO: The discussion shifts to GPU-level optimizations. Linda details how the authors use Ping-Pong scheduling and asynchronous TMA on Hopper GPUs to hide memory latency behind matrix math.
• Token Rounding Routing: Professor Norris questions the padding waste in Grouped GEMMs. Linda reveals the paper's novel Token Rounding method that aligns token routing exactly with hardware tile sizes, saving massive compute.
• Conclusion and Impact: The hosts wrap up the episode by discussing the broader implications of SonicMoE for the future of large language model training and scaling laws.