Sign up to save your podcasts
Or
Share AIConfigurator for Cross-Framework LLM Serving
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AIConfigurator for Cross-Framework LLM Serving
This episode explores AIConfigurator, a NVIDIA-led system for optimizing LLM serving configurations across frameworks such as TensorRT-LLM, vLLM, SGLang, and NVIDIA’s internal stack without brute-force benchmarking. It explains why operators should care more about TTFT, TPOT, and goodput than raw tokens-per-second, and unpacks the real serving tradeoffs around prefill/decode disaggregation, hybrid tensor/pipeline/expert parallelism, CUDA graphs, KV-cache sizing, and token limits. The discussion argues that the paper’s main contribution is a calibrated, framework-agnostic performance model built from primitive costs like GEMMs, attention, communication, and memory operations, then combined with backend-specific scheduling behavior to search thousands of deployment choices quickly. It is especially interesting for listeners who want a concrete view of LLM deployment economics: how to translate hardware budgets and latency targets into practical, high-performing serving setups without wasting days of GPU tuning.
Sources:
1. AIConfigurator for Cross-Framework LLM Serving
https://arxiv.org/pdf/2601.06288
2. LLM Inference Serving: Survey of Recent Advances and Opportunities — Baolin Li, Yankai Jiang, Vijay Gadepally, Devesh Tiwari, 2024
https://scholar.google.com/scholar?q=LLM+Inference+Serving%3A+Survey+of+Recent+Advances+and+Opportunities
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
4. Vidur: A Large-Scale Simulation Framework For LLM Inference — Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, Alexey Tumanov, 2024
https://scholar.google.com/scholar?q=Vidur%3A+A+Large-Scale+Simulation+Framework+For+LLM+Inference
5. AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving — Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yiyi Chen, et al., 2026
https://scholar.google.com/scholar?q=AIConfigurator%3A+Lightning-Fast+Configuration+Optimization+for+Multi-Framework+LLM+Serving
6. Apex: An Extensible and Dynamism-aware Simulator for Automated Parallel Execution in LLM Serving — Yi-Chien Lin et al., 2024
https://scholar.google.com/scholar?q=Apex%3A+An+Extensible+and+Dynamism-aware+Simulator+for+Automated+Parallel+Execution+in+LLM+Serving
7. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong et al., 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
8. Splitwise: Efficient Generative LLM Inference using Phase Splitting — Pratyush Patel et al., 2024
https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+using+Phase+Splitting
9. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin et al., 2024
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
10. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal et al., 2024
https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve
11. KVShare: Semantic-Aware Key-Value Cache Sharing for Efficient Large Language Model Inference — Huan Yang et al., 2025
https://arxiv.org/abs/2503.16525
12. Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management — Haoyu Zheng et al., 2026
https://arxiv.org/abs/2605.06472
13. PyGraph: Robust Compiler Support for CUDA Graphs in PyTorch — Abhishek Ghosh et al., 2025
https://arxiv.org/abs/2503.19779
14. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving — Chao Wang et al., 2025
https://arxiv.org/abs/2508.01989
15. Enhancing LLM Efficiency: Targeted Pruning for Prefill-Decode Disaggregation in Inference — Hao Zhang et al., 2025
https://arxiv.org/abs/2509.04467
16. Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts — Xuan-Phi Nguyen et al., 2026
https://arxiv.org/abs/2601.17111
17. AI Post Transformers: LLMServingSim 2.0 for Disaggregated LLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-27-llmservingsim-20-for-disaggregated-llm-s-05c04b.mp3
18. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp3
19. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3
20. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp3
View all episodes
By mcgrofThis episode explores AIConfigurator, a NVIDIA-led system for optimizing LLM serving configurations across frameworks such as TensorRT-LLM, vLLM, SGLang, and NVIDIA’s internal stack without brute-force benchmarking. It explains why operators should care more about TTFT, TPOT, and goodput than raw tokens-per-second, and unpacks the real serving tradeoffs around prefill/decode disaggregation, hybrid tensor/pipeline/expert parallelism, CUDA graphs, KV-cache sizing, and token limits. The discussion argues that the paper’s main contribution is a calibrated, framework-agnostic performance model built from primitive costs like GEMMs, attention, communication, and memory operations, then combined with backend-specific scheduling behavior to search thousands of deployment choices quickly. It is especially interesting for listeners who want a concrete view of LLM deployment economics: how to translate hardware budgets and latency targets into practical, high-performing serving setups without wasting days of GPU tuning.
Sources:
1. AIConfigurator for Cross-Framework LLM Serving
https://arxiv.org/pdf/2601.06288
2. LLM Inference Serving: Survey of Recent Advances and Opportunities — Baolin Li, Yankai Jiang, Vijay Gadepally, Devesh Tiwari, 2024
https://scholar.google.com/scholar?q=LLM+Inference+Serving%3A+Survey+of+Recent+Advances+and+Opportunities
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
4. Vidur: A Large-Scale Simulation Framework For LLM Inference — Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, Alexey Tumanov, 2024
https://scholar.google.com/scholar?q=Vidur%3A+A+Large-Scale+Simulation+Framework+For+LLM+Inference
5. AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving — Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yiyi Chen, et al., 2026
https://scholar.google.com/scholar?q=AIConfigurator%3A+Lightning-Fast+Configuration+Optimization+for+Multi-Framework+LLM+Serving
6. Apex: An Extensible and Dynamism-aware Simulator for Automated Parallel Execution in LLM Serving — Yi-Chien Lin et al., 2024
https://scholar.google.com/scholar?q=Apex%3A+An+Extensible+and+Dynamism-aware+Simulator+for+Automated+Parallel+Execution+in+LLM+Serving
7. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong et al., 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
8. Splitwise: Efficient Generative LLM Inference using Phase Splitting — Pratyush Patel et al., 2024
https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+using+Phase+Splitting
9. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin et al., 2024
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
10. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal et al., 2024
https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve
11. KVShare: Semantic-Aware Key-Value Cache Sharing for Efficient Large Language Model Inference — Huan Yang et al., 2025
https://arxiv.org/abs/2503.16525
12. Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management — Haoyu Zheng et al., 2026
https://arxiv.org/abs/2605.06472
13. PyGraph: Robust Compiler Support for CUDA Graphs in PyTorch — Abhishek Ghosh et al., 2025
https://arxiv.org/abs/2503.19779
14. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving — Chao Wang et al., 2025
https://arxiv.org/abs/2508.01989
15. Enhancing LLM Efficiency: Targeted Pruning for Prefill-Decode Disaggregation in Inference — Hao Zhang et al., 2025
https://arxiv.org/abs/2509.04467
16. Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts — Xuan-Phi Nguyen et al., 2026
https://arxiv.org/abs/2601.17111
17. AI Post Transformers: LLMServingSim 2.0 for Disaggregated LLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-27-llmservingsim-20-for-disaggregated-llm-s-05c04b.mp3
18. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp3
19. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3
20. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp3