Episode title: The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse Attention

A two-paper deep dive framed around the Dwarkesh Patel x Reiner Pope

blackboard lecture on training and serving frontier LLMs. The hosts

work through "Unified Scaling Laws for Routed Language Models" (Clark

et al., DeepMind 2022, arXiv 2202.01169) for the mixture-of-experts

side and the DeepSeek sparse-attention paper (arXiv 2512.02556) for

the attention side, treating Pope's blackboard framing on the

podcast as the pedagogical lens. The episode separates what the

papers establish from what Pope's practitioner intuition adds on

top, with particular attention to how MoE on the FFN side and sparse

attention on the QK side attack independent cost pools and can

compound rather than compete.

Sources:

arXiv 2202.01169 — "Unified Scaling Laws for Routed Language Models"

https://arxiv.org/pdf/2202.01169

arXiv 2512.02556 — DeepSeek sparse-attention paper

https://arxiv.org/pdf/2512.02556

Dwarkesh Podcast — "Reiner Pope: The math behind how LLMs are trained and served" (April 29 2026)

https://www.dwarkesh.com/p/reiner-pope

Transcript: https://gist.github.com/dwarkeshsp/79100f0fdeed69d76241903bb0604dbe

Older MoE context: GShard (arXiv 2006.16668), Switch Transformer (arXiv 2101.03961)

Chinchilla scaling laws (arXiv 2203.15556) — referenced in the Pope episode

Interactive Visualization: The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse Attention