Sign up to save your podcasts
Or
Share Systematic Characterization of LLM Inference on GPUs
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Systematic Characterization of LLM Inference on GPUs
Hal Turing and Dr. Ada Shannon examine "A Systematic Characterization of LLM Inference on GPUs" by Haonan Wang and colleagues from the Institute of Computing Technology at the Chinese Academy of Sciences, Zhejiang Lab, China Telecom, and Peking University. Dr. Shannon opens by framing the paper's central contribution: the inference literature is fragmented across power consumption studies, kernel profiling, quantization analysis, and serving schedulers, none sharing a common analytical vocabulary. Wang et al. attempt to bridge ML systems thinking with computer architecture rigor into a unified cross-layer framework — from "the model feels slow" down to the specific hardware bottleneck and its root cause.
The hosts build the foundational model carefully, tracing execution from prompt to generated token. Dr. Shannon explains the two-phase structure of transformer inference: the prefill phase processes all prompt tokens simultaneously in a matrix-matrix multiply that is compute-bound, writing key-value pairs into the KV cache; the decode phase generates tokens autoregressively, and at each step the attention mechanism reads back every prior cached key-value pair alongside the full weight matrices. Hal presses on the memory bandwidth implications — by token three hundred and ninety-nine, the model loads nearly four hundred KV pairs per layer per step just to produce one new token. Dr. Shannon maps this onto the Roofline model, originally developed by Williams, Waterman, and Patterson at UC Berkeley in 2009, using arithmetic intensity — the ratio of floating-point operations to memory traffic — to show why decode sits deep in the memory-bound regime while prefill remains compute-bound. That asymmetry is the load-bearing structure of the paper's argument.
When Hal pushes back on whether the compute-bound prefill and memory-bound decode framing is merely a formalization of established practitioner intuition, Dr. Shannon draws a sharp distinction between the heuristic and the science. Knowing decode is memory-bound is the starting point; the paper's contribution is stall analysis — profiling with hardware performance counters to determine precisely why GPU thread warps pause during execution. A warp stalled on memory pipe saturation has a different cause and a different remedy than one stalled on instruction-level data dependency. The hosts frame this microarchitectural root cause analysis as one of four dimensions in the paper's framework, alongside the two-phase prefill-decode heterogeneity, system scaling principles, and the boundaries where current GPU architectures hit fundamental limits — together forming the diagnostic vocabulary the field has been missing.
Sources:
1. A Systematic Characterization of LLM Inference on GPUs — Haonan Wang, Xuxin Xiao, Mingyu Yan, Zhuoyuan Zhu, Dengke Han, Duo Wang, Wenming Li, Xiaochun Ye, Cunchen Hu, Hongyang Chen, Guangyu Sun, 2025
http://arxiv.org/abs/2512.01644
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
3. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun, 2022
https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models
4. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving — Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xing Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, Ion Stoica, 2022
https://scholar.google.com/scholar?q=AlpaServe%3A+Statistical+Multiplexing+with+Model+Parallelism+for+Deep+Learning+Serving
5. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022
https://scholar.google.com/scholar?q=LLM.int8%28%29%3A+8-bit+Matrix+Multiplication+for+Transformers+at+Scale
6. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini, 2024
https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting
7. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+Large+Language+Model+Serving
8. Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes on Chunked Prefills — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024
https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+on+Chunked+Prefills
9. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang, 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
10. Roofline: An Insightful Visual Performance Model for Multicore Architectures — Samuel Williams, Andrew Waterman, David Patterson, 2009
https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Multicore+Architectures
11. In-Datacenter Performance Analysis of a Tensor Processing Unit — Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, et al., 2017
https://scholar.google.com/scholar?q=In-Datacenter+Performance+Analysis+of+a+Tensor+Processing+Unit
12. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022
https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness
13. Dissecting the Graphcore IPU Architecture via Microbenchmarking — Amey Agrawal, Rohit Ramaprasad, Kranthi Gade, Ramachandran Ramjee, 2021
https://scholar.google.com/scholar?q=Dissecting+the+Graphcore+IPU+Architecture+via+Microbenchmarking
14. Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Agrawal et al., 2024
https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills
15. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Dao, 2023
https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning
16. Flash-Decoding for Long-Context Inference — Dao et al., 2023
https://scholar.google.com/scholar?q=Flash-Decoding+for+Long-Context+Inference
17. Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures — Williams, Waterman, Patterson, 2009
https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Floating-Point+Programs+and+Multicore+Architectures
18. Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Agrawal et al., 2023
https://scholar.google.com/scholar?q=Sarathi%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills
19. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=POD-Attention%3A+Unlocking+Full+Prefill-Decode+Overlap+for+Faster+LLM+Inference
20. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=KVLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse
21. Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=Shared+Disk+KV+Cache+Management+for+Efficient+Multi-Instance+Inference+in+RAG-Powered+LLMs
22. ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=ExpertFlow%3A+Optimized+Expert+Activation+and+Token+Allocation+for+Efficient+Mixture-of-Experts+Inference
23. Harder Task Needs More Experts: Dynamic Routing in MoE Models — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=Harder+Task+Needs+More+Experts%3A+Dynamic+Routing+in+MoE+Models
24. DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=DECA%3A+A+Near-Core+LLM+Decompression+Accelerator+Grounded+on+a+3D+Roofline+Model
25. HeadInfer: Memory-Efficient LLM Inference by Head-Wise Offloading — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=HeadInfer%3A+Memory-Efficient+LLM+Inference+by+Head-Wise+Offloading
26. AI Post Transformers: AI and the Memory Wall: Overcoming Bottlenecks — Hal Turing & Dr. Ada Shannon, Fri,
https://podcasters.spotify.com/pod/show/12146088098/episodes/AI-and-the-Memory-Wall-Overcoming-Bottlenecks-e36ki0u
27. AI Post Transformers: Flash-LLM: Efficient LLM Inference with Unstructured Sparsity on Tensor Cores — Hal Turing & Dr. Ada Shannon, Fri,
https://podcasters.spotify.com/pod/show/12146088098/episodes/Flash-LLM-Efficient-LLM-Inference-with-Unstructured-Sparsity-on-Tensor-Cores-e3aalrl
28. AI Post Transformers: Advancements in Efficient KV Cache Quantization and Management — Hal Turing & Dr. Ada Shannon, Thu,
https://podcasters.spotify.com/pod/show/12146088098/episodes/Advancements-in-Efficient-KV-Cache-Quantization-and-Management-e3fk9kr
29. AI Post Transformers: Adaptive LLM Partitioning for Edge Inference — Hal Turing & Dr. Ada Shannon, Tue,
https://podcasters.spotify.com/pod/show/12146088098/episodes/Adaptive-LLM-Partitioning-for-Edge-Inference-e38a9mn
30. AI Post Transformers: Parallel Token Prediction: From ProphetNet to Dependent Multi-Token Generation — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-04-4-papers-combined-8a237d.mp3
Interactive Visualization: Systematic Characterization of LLM Inference on GPUs
View all episodes
By mcgrofHal Turing and Dr. Ada Shannon examine "A Systematic Characterization of LLM Inference on GPUs" by Haonan Wang and colleagues from the Institute of Computing Technology at the Chinese Academy of Sciences, Zhejiang Lab, China Telecom, and Peking University. Dr. Shannon opens by framing the paper's central contribution: the inference literature is fragmented across power consumption studies, kernel profiling, quantization analysis, and serving schedulers, none sharing a common analytical vocabulary. Wang et al. attempt to bridge ML systems thinking with computer architecture rigor into a unified cross-layer framework — from "the model feels slow" down to the specific hardware bottleneck and its root cause.
The hosts build the foundational model carefully, tracing execution from prompt to generated token. Dr. Shannon explains the two-phase structure of transformer inference: the prefill phase processes all prompt tokens simultaneously in a matrix-matrix multiply that is compute-bound, writing key-value pairs into the KV cache; the decode phase generates tokens autoregressively, and at each step the attention mechanism reads back every prior cached key-value pair alongside the full weight matrices. Hal presses on the memory bandwidth implications — by token three hundred and ninety-nine, the model loads nearly four hundred KV pairs per layer per step just to produce one new token. Dr. Shannon maps this onto the Roofline model, originally developed by Williams, Waterman, and Patterson at UC Berkeley in 2009, using arithmetic intensity — the ratio of floating-point operations to memory traffic — to show why decode sits deep in the memory-bound regime while prefill remains compute-bound. That asymmetry is the load-bearing structure of the paper's argument.
When Hal pushes back on whether the compute-bound prefill and memory-bound decode framing is merely a formalization of established practitioner intuition, Dr. Shannon draws a sharp distinction between the heuristic and the science. Knowing decode is memory-bound is the starting point; the paper's contribution is stall analysis — profiling with hardware performance counters to determine precisely why GPU thread warps pause during execution. A warp stalled on memory pipe saturation has a different cause and a different remedy than one stalled on instruction-level data dependency. The hosts frame this microarchitectural root cause analysis as one of four dimensions in the paper's framework, alongside the two-phase prefill-decode heterogeneity, system scaling principles, and the boundaries where current GPU architectures hit fundamental limits — together forming the diagnostic vocabulary the field has been missing.
Sources:
1. A Systematic Characterization of LLM Inference on GPUs — Haonan Wang, Xuxin Xiao, Mingyu Yan, Zhuoyuan Zhu, Dengke Han, Duo Wang, Wenming Li, Xiaochun Ye, Cunchen Hu, Hongyang Chen, Guangyu Sun, 2025
http://arxiv.org/abs/2512.01644
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
3. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun, 2022
https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models
4. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving — Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xing Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, Ion Stoica, 2022
https://scholar.google.com/scholar?q=AlpaServe%3A+Statistical+Multiplexing+with+Model+Parallelism+for+Deep+Learning+Serving
5. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022
https://scholar.google.com/scholar?q=LLM.int8%28%29%3A+8-bit+Matrix+Multiplication+for+Transformers+at+Scale
6. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini, 2024
https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting
7. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+Large+Language+Model+Serving
8. Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes on Chunked Prefills — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024
https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+on+Chunked+Prefills
9. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang, 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
10. Roofline: An Insightful Visual Performance Model for Multicore Architectures — Samuel Williams, Andrew Waterman, David Patterson, 2009
https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Multicore+Architectures
11. In-Datacenter Performance Analysis of a Tensor Processing Unit — Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, et al., 2017
https://scholar.google.com/scholar?q=In-Datacenter+Performance+Analysis+of+a+Tensor+Processing+Unit
12. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022
https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness
13. Dissecting the Graphcore IPU Architecture via Microbenchmarking — Amey Agrawal, Rohit Ramaprasad, Kranthi Gade, Ramachandran Ramjee, 2021
https://scholar.google.com/scholar?q=Dissecting+the+Graphcore+IPU+Architecture+via+Microbenchmarking
14. Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Agrawal et al., 2024
https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills
15. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Dao, 2023
https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning
16. Flash-Decoding for Long-Context Inference — Dao et al., 2023
https://scholar.google.com/scholar?q=Flash-Decoding+for+Long-Context+Inference
17. Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures — Williams, Waterman, Patterson, 2009
https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Floating-Point+Programs+and+Multicore+Architectures
18. Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Agrawal et al., 2023
https://scholar.google.com/scholar?q=Sarathi%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills
19. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=POD-Attention%3A+Unlocking+Full+Prefill-Decode+Overlap+for+Faster+LLM+Inference
20. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=KVLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse
21. Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=Shared+Disk+KV+Cache+Management+for+Efficient+Multi-Instance+Inference+in+RAG-Powered+LLMs
22. ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=ExpertFlow%3A+Optimized+Expert+Activation+and+Token+Allocation+for+Efficient+Mixture-of-Experts+Inference
23. Harder Task Needs More Experts: Dynamic Routing in MoE Models — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=Harder+Task+Needs+More+Experts%3A+Dynamic+Routing+in+MoE+Models
24. DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=DECA%3A+A+Near-Core+LLM+Decompression+Accelerator+Grounded+on+a+3D+Roofline+Model
25. HeadInfer: Memory-Efficient LLM Inference by Head-Wise Offloading — approximate, recent, 2024-2025
https://scholar.google.com/scholar?q=HeadInfer%3A+Memory-Efficient+LLM+Inference+by+Head-Wise+Offloading
26. AI Post Transformers: AI and the Memory Wall: Overcoming Bottlenecks — Hal Turing & Dr. Ada Shannon, Fri,
https://podcasters.spotify.com/pod/show/12146088098/episodes/AI-and-the-Memory-Wall-Overcoming-Bottlenecks-e36ki0u
27. AI Post Transformers: Flash-LLM: Efficient LLM Inference with Unstructured Sparsity on Tensor Cores — Hal Turing & Dr. Ada Shannon, Fri,
https://podcasters.spotify.com/pod/show/12146088098/episodes/Flash-LLM-Efficient-LLM-Inference-with-Unstructured-Sparsity-on-Tensor-Cores-e3aalrl
28. AI Post Transformers: Advancements in Efficient KV Cache Quantization and Management — Hal Turing & Dr. Ada Shannon, Thu,
https://podcasters.spotify.com/pod/show/12146088098/episodes/Advancements-in-Efficient-KV-Cache-Quantization-and-Management-e3fk9kr
29. AI Post Transformers: Adaptive LLM Partitioning for Edge Inference — Hal Turing & Dr. Ada Shannon, Tue,
https://podcasters.spotify.com/pod/show/12146088098/episodes/Adaptive-LLM-Partitioning-for-Edge-Inference-e38a9mn
30. AI Post Transformers: Parallel Token Prediction: From ProphetNet to Dependent Multi-Token Generation — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-04-4-papers-combined-8a237d.mp3
Interactive Visualization: Systematic Characterization of LLM Inference on GPUs