Sign up to save your podcasts
Or
Share xLLM: Co-Locating Online and Offline LLM Inference
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
xLLM: Co-Locating Online and Offline LLM Inference
This episode explores the xLLM Technical Report from JD.com, which describes a production system for running large language model inference at enterprise scale. The core technical challenge is co-locating online chatbot workloads with offline batch jobs on the same GPU cluster to maximize hardware utilization during traffic valleys, while maintaining strict latency guarantees during peak hours. The discussion covers key architectural decisions including dynamic prefill-decode disaggregation, which adaptively reallocates compute resources between the prompt processing phase and token generation phase based on real-time workload characteristics, and the management of unpredictable KV cache memory growth that makes LLM co-location harder than traditional cloud workload mixing. Listeners interested in production ML systems engineering, GPU cluster optimization, and the practical challenges of deploying transformer models at scale will find concrete insights into how a major tech company handles the resource scheduling problems that academic papers often overlook.
Sources:
1. xLLM Technical Report — Tongxuan Liu, Tao Peng, Peijun Yang, Xiaoyang Zhao, Xiusheng Lu, Weizhe Huang, Zirui Liu, Xiaoyu Chen, Zhiwei Liang, Jun Xiong, Donghe Jin, Minchao Zhang, Jinrong Guo, Yingxu Deng, Xu Zhang, Xianzhe Dong, Siqi Wang, Siyu Wu, Yu Wu, Zihan Tang, Yuting Zeng, Yanshu Wang, Jinguang Liu, Meng Kang, Menxin Li, Yunlong Wang, Yiming Liu, Xiaolong Ma, Yifan Wang, Yichen Zhang, Jinrun Yin, Keyang Zheng, Jiawei Yin, Jun Zhang, Ziyue Wang, Xiaobo Lin, Liangyu Liu, Liwei Lan, Yang Liu, Chunhua Peng, Han Liu, Songcheng Ren, Xuezhu Wang, Yunheng Shen, Yi Wang, Guyue Liu, Yitao Hu, Hui Chen, Tong Yang, Hailong Yang, Jing Li, Guiguang Ding, Ke Zhang, 2025
http://arxiv.org/abs/2510.14686v2
2. Borg, Omega, and Kubernetes: Lessons learned from three container-management systems over a decade — Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer, John Wilkes, 2016
https://scholar.google.com/scholar?q=Borg%2C+Omega%2C+and+Kubernetes%3A+Lessons+learned+from+three+container-management+systems+over+a+decade
3. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, 2023
https://scholar.google.com/scholar?q=SARATHI%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills
4. Paella: Low-latency Model Serving with Virtualized GPU Scheduling — Yiying Zhang, et al., 2024
https://scholar.google.com/scholar?q=Paella%3A+Low-latency+Model+Serving+with+Virtualized+GPU+Scheduling
5. Synergy: A QoS-aware Multi-DNN Inference Framework for Heterogeneous Systems-on-Chip — Jie Ren, et al., 2022
https://scholar.google.com/scholar?q=Synergy%3A+A+QoS-aware+Multi-DNN+Inference+Framework+for+Heterogeneous+Systems-on-Chip
6. TensorFlow: A System for Large-Scale Machine Learning — Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, et al., 2016
https://scholar.google.com/scholar?q=TensorFlow%3A+A+System+for+Large-Scale+Machine+Learning
7. XLA: Optimizing Compiler for Machine Learning — The TensorFlow team (Google), 2017
https://scholar.google.com/scholar?q=XLA%3A+Optimizing+Compiler+for+Machine+Learning
8. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, et al., 2018
https://scholar.google.com/scholar?q=TVM%3A+An+Automated+End-to-End+Optimizing+Compiler+for+Deep+Learning
9. torch.compile: A Compiler for PyTorch Models (PyTorch 2.0 Technical Report) — Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, et al. (Meta), 2023
https://scholar.google.com/scholar?q=torch.compile%3A+A+Compiler+for+PyTorch+Models+%28PyTorch+2.0+Technical+Report%29
10. Orca: A Distributed Serving System for Transformer-Based Generative Models — Yu et al., 2022
https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models
11. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Sheng et al., 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
12. DeepSpeed-MII: Instant Speedup on Inference with Advanced Model Implementations and System Techniques — Microsoft, 2023
https://scholar.google.com/scholar?q=DeepSpeed-MII%3A+Instant+Speedup+on+Inference+with+Advanced+Model+Implementations+and+System+Techniques
13. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving — Li et al., 2023
https://scholar.google.com/scholar?q=AlpaServe%3A+Statistical+Multiplexing+with+Model+Parallelism+for+Deep+Learning+Serving
14. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Cai et al., 2024
https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads
15. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference — approximate 2025, 2025
https://scholar.google.com/scholar?q=POD-Attention%3A+Unlocking+Full+Prefill-Decode+Overlap+for+Faster+LLM+Inference
16. FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving — approximate 2025, 2025
https://scholar.google.com/scholar?q=FlowPrefill%3A+Decoupling+Preemption+from+Prefill+Scheduling+Granularity+to+Mitigate+Head-of-Line+Blocking+in+LLM+Serving
17. KvLink: Accelerating Large Language Models via Efficient KV Cache Reuse — approximate 2024-2025, 2024
https://scholar.google.com/scholar?q=KvLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse
18. Decoding Speculative Decoding — approximate 2024, 2024
https://scholar.google.com/scholar?q=Decoding+Speculative+Decoding
19. A Comprehensive Survey of Mixture-of-Experts — approximate 2024-2025, 2024
https://scholar.google.com/scholar?q=A+Comprehensive+Survey+of+Mixture-of-Experts
20. DualPath Breaks Storage Bandwidth Bottleneck in Agentic Inference
https://podcast.do-not-panic.com/episodes/2026-03-07-dualpath-breaks-storage-bandwidth-bottle-bc9a82.mp3
21. FlashAttention-4 Conquers Asymmetric GPU Hardware Scaling
https://podcast.do-not-panic.com/episodes/2026-03-06-flashattention-4-conquers-asymmetric-gpu-78839b.mp3
22. 50x KV Cache Compression in Seconds via Attention Matching
https://podcast.do-not-panic.com/episodes/2026-03-09-50x-kv-cache-compression-in-seconds-via-9402c1.mp3
23. Systematic Characterization of LLM Inference on GPUs
https://podcast.do-not-panic.com/episodes/2026-03-07-systematic-characterization-of-llm-infer-f7b9a8.mp3
Interactive Visualization: xLLM: Co-Locating Online and Offline LLM Inference
View all episodes
By mcgrofThis episode explores the xLLM Technical Report from JD.com, which describes a production system for running large language model inference at enterprise scale. The core technical challenge is co-locating online chatbot workloads with offline batch jobs on the same GPU cluster to maximize hardware utilization during traffic valleys, while maintaining strict latency guarantees during peak hours. The discussion covers key architectural decisions including dynamic prefill-decode disaggregation, which adaptively reallocates compute resources between the prompt processing phase and token generation phase based on real-time workload characteristics, and the management of unpredictable KV cache memory growth that makes LLM co-location harder than traditional cloud workload mixing. Listeners interested in production ML systems engineering, GPU cluster optimization, and the practical challenges of deploying transformer models at scale will find concrete insights into how a major tech company handles the resource scheduling problems that academic papers often overlook.
Sources:
1. xLLM Technical Report — Tongxuan Liu, Tao Peng, Peijun Yang, Xiaoyang Zhao, Xiusheng Lu, Weizhe Huang, Zirui Liu, Xiaoyu Chen, Zhiwei Liang, Jun Xiong, Donghe Jin, Minchao Zhang, Jinrong Guo, Yingxu Deng, Xu Zhang, Xianzhe Dong, Siqi Wang, Siyu Wu, Yu Wu, Zihan Tang, Yuting Zeng, Yanshu Wang, Jinguang Liu, Meng Kang, Menxin Li, Yunlong Wang, Yiming Liu, Xiaolong Ma, Yifan Wang, Yichen Zhang, Jinrun Yin, Keyang Zheng, Jiawei Yin, Jun Zhang, Ziyue Wang, Xiaobo Lin, Liangyu Liu, Liwei Lan, Yang Liu, Chunhua Peng, Han Liu, Songcheng Ren, Xuezhu Wang, Yunheng Shen, Yi Wang, Guyue Liu, Yitao Hu, Hui Chen, Tong Yang, Hailong Yang, Jing Li, Guiguang Ding, Ke Zhang, 2025
http://arxiv.org/abs/2510.14686v2
2. Borg, Omega, and Kubernetes: Lessons learned from three container-management systems over a decade — Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer, John Wilkes, 2016
https://scholar.google.com/scholar?q=Borg%2C+Omega%2C+and+Kubernetes%3A+Lessons+learned+from+three+container-management+systems+over+a+decade
3. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, 2023
https://scholar.google.com/scholar?q=SARATHI%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills
4. Paella: Low-latency Model Serving with Virtualized GPU Scheduling — Yiying Zhang, et al., 2024
https://scholar.google.com/scholar?q=Paella%3A+Low-latency+Model+Serving+with+Virtualized+GPU+Scheduling
5. Synergy: A QoS-aware Multi-DNN Inference Framework for Heterogeneous Systems-on-Chip — Jie Ren, et al., 2022
https://scholar.google.com/scholar?q=Synergy%3A+A+QoS-aware+Multi-DNN+Inference+Framework+for+Heterogeneous+Systems-on-Chip
6. TensorFlow: A System for Large-Scale Machine Learning — Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, et al., 2016
https://scholar.google.com/scholar?q=TensorFlow%3A+A+System+for+Large-Scale+Machine+Learning
7. XLA: Optimizing Compiler for Machine Learning — The TensorFlow team (Google), 2017
https://scholar.google.com/scholar?q=XLA%3A+Optimizing+Compiler+for+Machine+Learning
8. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, et al., 2018
https://scholar.google.com/scholar?q=TVM%3A+An+Automated+End-to-End+Optimizing+Compiler+for+Deep+Learning
9. torch.compile: A Compiler for PyTorch Models (PyTorch 2.0 Technical Report) — Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, et al. (Meta), 2023
https://scholar.google.com/scholar?q=torch.compile%3A+A+Compiler+for+PyTorch+Models+%28PyTorch+2.0+Technical+Report%29
10. Orca: A Distributed Serving System for Transformer-Based Generative Models — Yu et al., 2022
https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models
11. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Sheng et al., 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
12. DeepSpeed-MII: Instant Speedup on Inference with Advanced Model Implementations and System Techniques — Microsoft, 2023
https://scholar.google.com/scholar?q=DeepSpeed-MII%3A+Instant+Speedup+on+Inference+with+Advanced+Model+Implementations+and+System+Techniques
13. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving — Li et al., 2023
https://scholar.google.com/scholar?q=AlpaServe%3A+Statistical+Multiplexing+with+Model+Parallelism+for+Deep+Learning+Serving
14. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Cai et al., 2024
https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads
15. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference — approximate 2025, 2025
https://scholar.google.com/scholar?q=POD-Attention%3A+Unlocking+Full+Prefill-Decode+Overlap+for+Faster+LLM+Inference
16. FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving — approximate 2025, 2025
https://scholar.google.com/scholar?q=FlowPrefill%3A+Decoupling+Preemption+from+Prefill+Scheduling+Granularity+to+Mitigate+Head-of-Line+Blocking+in+LLM+Serving
17. KvLink: Accelerating Large Language Models via Efficient KV Cache Reuse — approximate 2024-2025, 2024
https://scholar.google.com/scholar?q=KvLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse
18. Decoding Speculative Decoding — approximate 2024, 2024
https://scholar.google.com/scholar?q=Decoding+Speculative+Decoding
19. A Comprehensive Survey of Mixture-of-Experts — approximate 2024-2025, 2024
https://scholar.google.com/scholar?q=A+Comprehensive+Survey+of+Mixture-of-Experts
20. DualPath Breaks Storage Bandwidth Bottleneck in Agentic Inference
https://podcast.do-not-panic.com/episodes/2026-03-07-dualpath-breaks-storage-bandwidth-bottle-bc9a82.mp3
21. FlashAttention-4 Conquers Asymmetric GPU Hardware Scaling
https://podcast.do-not-panic.com/episodes/2026-03-06-flashattention-4-conquers-asymmetric-gpu-78839b.mp3
22. 50x KV Cache Compression in Seconds via Attention Matching
https://podcast.do-not-panic.com/episodes/2026-03-09-50x-kv-cache-compression-in-seconds-via-9402c1.mp3
23. Systematic Characterization of LLM Inference on GPUs
https://podcast.do-not-panic.com/episodes/2026-03-07-systematic-characterization-of-llm-infer-f7b9a8.mp3
Interactive Visualization: xLLM: Co-Locating Online and Offline LLM Inference