Sign up to save your podcasts
Or
Share TensorFlow for Distributed Machine Learning Systems
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
TensorFlow for Distributed Machine Learning Systems
This episode explores the TensorFlow paper as a systems argument for unifying the full machine learning lifecycle, from mobile inference to large-scale distributed training, within a single stateful dataflow framework. It explains how TensorFlow represents computation as graphs with mutable state, why that mattered for device placement, parameter storage, checkpointing, and heterogeneous hardware, and how it aimed to improve on the limitations of DistBelief. The discussion also places the paper in the broader lineage of MapReduce, Dryad, Naiad, and parameter-server training, while debating whether TensorFlow truly generalized machine learning workflows or mainly fit the kinds of static, graph-friendly workloads large organizations like Google already needed. Listeners would find it interesting for its mix of technical history, distributed systems insight, and a clear-eyed look at the tradeoff between organizational scale, portability, and usability for everyday researchers.
Sources:
1. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems — Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, Xiaoqiang Zheng, 2016
http://arxiv.org/abs/1603.04467
2. MapReduce: Simplified Data Processing on Large Clusters — Jeffrey Dean and Sanjay Ghemawat, 2004
https://scholar.google.com/scholar?q=MapReduce%3A+Simplified+Data+Processing+on+Large+Clusters
3. Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks — Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly, 2007
https://scholar.google.com/scholar?q=Dryad%3A+Distributed+Data-Parallel+Programs+from+Sequential+Building+Blocks
4. Large Scale Distributed Deep Networks — Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng, 2012
https://scholar.google.com/scholar?q=Large+Scale+Distributed+Deep+Networks
5. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems — Martín Abadi, Ashish Agarwal, Paul Barham, Jeffrey Dean, Rajat Monga, and many others, 2016
https://scholar.google.com/scholar?q=TensorFlow%3A+Large-Scale+Machine+Learning+on+Heterogeneous+Distributed+Systems
6. Naiad: A Timely Dataflow System — Frank McSherry, Derek G. Murray, Rebecca Isaacs, and Michael Isard, 2013
https://scholar.google.com/scholar?q=Naiad%3A+A+Timely+Dataflow+System
7. Project Adam: Building an Efficient and Scalable Deep Learning Training System — Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman, 2014
https://scholar.google.com/scholar?q=Project+Adam%3A+Building+an+Efficient+and+Scalable+Deep+Learning+Training+System
8. Parameter Server for Distributed Machine Learning — Mu Li, David G. Andersen, Alexander J. Smola, and Kai Yu, 2014
https://scholar.google.com/scholar?q=Parameter+Server+for+Distributed+Machine+Learning
9. Caffe: Convolutional Architecture for Fast Feature Embedding — Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell, 2014
https://scholar.google.com/scholar?q=Caffe%3A+Convolutional+Architecture+for+Fast+Feature+Embedding
10. Theano: A CPU and GPU Math Compiler in Python — James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio, 2010
https://scholar.google.com/scholar?q=Theano%3A+A+CPU+and+GPU+Math+Compiler+in+Python
11. MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems — Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang, 2015
https://scholar.google.com/scholar?q=MXNet%3A+A+Flexible+and+Efficient+Machine+Learning+Library+for+Heterogeneous+Distributed+Systems
12. SIMPLE: Efficient Temporal Graph Neural Network Training at Scale with Dynamic Data Placement — Shihong Gao, Yiming Li, Xin Zhang, Yanyan Shen, Yingxia Shao, Lei Chen, 2024
https://scholar.google.com/scholar?q=SIMPLE%3A+Efficient+Temporal+Graph+Neural+Network+Training+at+Scale+with+Dynamic+Data+Placement
13. Strategy-Switch: From All-Reduce to Parameter Server for Faster Efficient Training — Nikodimos Provatas, Iasonas Chalas, Ioannis Konstantinou, Nectarios Koziris, 2025
https://scholar.google.com/scholar?q=Strategy-Switch%3A+From+All-Reduce+to+Parameter+Server+for+Faster+Efficient+Training
14. Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models — Longteng Zhang, Xiang Liu, Zeyu Li, Xinglin Pan, Peijie Dong, Ruibo Fan, Rui Guo, Xin Wang, Qiong Luo, Shaohuai Shi, Xiaowen Chu, 2023
https://scholar.google.com/scholar?q=Dissecting+the+Runtime+Performance+of+the+Training%2C+Fine-tuning%2C+and+Inference+of+Large+Language+Models
15. AI Post Transformers: ONNX Ecosystem, Optimization, and Deployment — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/onnx-ecosystem-optimization-and-deployment/
Interactive Visualization: TensorFlow for Distributed Machine Learning Systems
View all episodes
By mcgrofThis episode explores the TensorFlow paper as a systems argument for unifying the full machine learning lifecycle, from mobile inference to large-scale distributed training, within a single stateful dataflow framework. It explains how TensorFlow represents computation as graphs with mutable state, why that mattered for device placement, parameter storage, checkpointing, and heterogeneous hardware, and how it aimed to improve on the limitations of DistBelief. The discussion also places the paper in the broader lineage of MapReduce, Dryad, Naiad, and parameter-server training, while debating whether TensorFlow truly generalized machine learning workflows or mainly fit the kinds of static, graph-friendly workloads large organizations like Google already needed. Listeners would find it interesting for its mix of technical history, distributed systems insight, and a clear-eyed look at the tradeoff between organizational scale, portability, and usability for everyday researchers.
Sources:
1. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems — Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, Xiaoqiang Zheng, 2016
http://arxiv.org/abs/1603.04467
2. MapReduce: Simplified Data Processing on Large Clusters — Jeffrey Dean and Sanjay Ghemawat, 2004
https://scholar.google.com/scholar?q=MapReduce%3A+Simplified+Data+Processing+on+Large+Clusters
3. Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks — Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly, 2007
https://scholar.google.com/scholar?q=Dryad%3A+Distributed+Data-Parallel+Programs+from+Sequential+Building+Blocks
4. Large Scale Distributed Deep Networks — Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng, 2012
https://scholar.google.com/scholar?q=Large+Scale+Distributed+Deep+Networks
5. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems — Martín Abadi, Ashish Agarwal, Paul Barham, Jeffrey Dean, Rajat Monga, and many others, 2016
https://scholar.google.com/scholar?q=TensorFlow%3A+Large-Scale+Machine+Learning+on+Heterogeneous+Distributed+Systems
6. Naiad: A Timely Dataflow System — Frank McSherry, Derek G. Murray, Rebecca Isaacs, and Michael Isard, 2013
https://scholar.google.com/scholar?q=Naiad%3A+A+Timely+Dataflow+System
7. Project Adam: Building an Efficient and Scalable Deep Learning Training System — Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman, 2014
https://scholar.google.com/scholar?q=Project+Adam%3A+Building+an+Efficient+and+Scalable+Deep+Learning+Training+System
8. Parameter Server for Distributed Machine Learning — Mu Li, David G. Andersen, Alexander J. Smola, and Kai Yu, 2014
https://scholar.google.com/scholar?q=Parameter+Server+for+Distributed+Machine+Learning
9. Caffe: Convolutional Architecture for Fast Feature Embedding — Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell, 2014
https://scholar.google.com/scholar?q=Caffe%3A+Convolutional+Architecture+for+Fast+Feature+Embedding
10. Theano: A CPU and GPU Math Compiler in Python — James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio, 2010
https://scholar.google.com/scholar?q=Theano%3A+A+CPU+and+GPU+Math+Compiler+in+Python
11. MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems — Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang, 2015
https://scholar.google.com/scholar?q=MXNet%3A+A+Flexible+and+Efficient+Machine+Learning+Library+for+Heterogeneous+Distributed+Systems
12. SIMPLE: Efficient Temporal Graph Neural Network Training at Scale with Dynamic Data Placement — Shihong Gao, Yiming Li, Xin Zhang, Yanyan Shen, Yingxia Shao, Lei Chen, 2024
https://scholar.google.com/scholar?q=SIMPLE%3A+Efficient+Temporal+Graph+Neural+Network+Training+at+Scale+with+Dynamic+Data+Placement
13. Strategy-Switch: From All-Reduce to Parameter Server for Faster Efficient Training — Nikodimos Provatas, Iasonas Chalas, Ioannis Konstantinou, Nectarios Koziris, 2025
https://scholar.google.com/scholar?q=Strategy-Switch%3A+From+All-Reduce+to+Parameter+Server+for+Faster+Efficient+Training
14. Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models — Longteng Zhang, Xiang Liu, Zeyu Li, Xinglin Pan, Peijie Dong, Ruibo Fan, Rui Guo, Xin Wang, Qiong Luo, Shaohuai Shi, Xiaowen Chu, 2023
https://scholar.google.com/scholar?q=Dissecting+the+Runtime+Performance+of+the+Training%2C+Fine-tuning%2C+and+Inference+of+Large+Language+Models
15. AI Post Transformers: ONNX Ecosystem, Optimization, and Deployment — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/onnx-ecosystem-optimization-and-deployment/
Interactive Visualization: TensorFlow for Distributed Machine Learning Systems