PyTorch Developer Podcast

What are complex numbers? What is conjugation? Why is conjugation so common in linear algebra? Why would we like conjugation to behave similarly to transposition (and why is matrix multiply with a transposed input so fast?) What is a conjugate view? How is it implemented? What's the relationship between views, laziness and call-by-name evaluation?

Further reading.

• Pull request that adds conjugate views https://github.com/pytorch/pytorch/pull/54987
• The idea of conjugate views originally came up when we were deciding which complex autograd convention to use in https://github.com/pytorch/pytorch/issues/41857 . PyTorch uses the conjugate Wirtinger derivative which, true to its name, involves a lot of conjugations in its formulas.
• Conjugate views are a form of bidirectional lens. This nice presentation explains what the concept is https://www.cis.upenn.edu/~bcpierce/papers/lenses-etapsslides.pdf

What historical constraints and design choices lead to the design of Tensor/Storage (and their Impl variants) as they are today? Why do we use intrusive refcounting? Why are we trying to get rid of virtual methods on TensorImpl? Why are there so many frickin' bitfields?

Further reading.

• PyTorch internals blog post http://blog.ezyang.com/2019/05/pytorch-internals/
• Writing Python in C++, a manifesto https://github.com/pytorch/pytorch/wiki/Writing-Python-in-cpp-(a-manifesto)
• At time of writing, the breakdown of all fields on TensorImpl https://github.com/pytorch/pytorch/blob/71f4c5c1f436258adc303b710efb3f41b2d50c4e/c10/core/TensorImpl.h#L2155-L2177

What's the general process by which a new operator is added to PyTorch? Why is this actually something of a rare occurrence? How do you integrate an operator with the rest of PyTorch's system so it can be run end-to-end? What should I expect if I'm writing a CPU and CUDA kernel? What tools are available to me to make the job easier? How can I debug my kernels? How do I test them?

Further reading.

• The README for the native/ directory, where all kernels get put https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/README.md
• A high level overview of how TensorIterator works https://labs.quansight.org/blog/2020/04/pytorch-tensoriterator-internals/
• Where OpInfos live https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/common_methods_invocations.py

What is a Variable? Why did it exist as a wrapper in the first place? Why did it get removed? How did we remove it? What are some of the lingering consequences of its removal?

Further reading:

• The release notes of PyTorch 0.4 do a good job explaining the user visible consequences of the removal, at the time, including how we "simulate" concepts on Variable that don't make sense anymore https://pytorch.org/blog/pytorch-0_4_0-migration-guide/
• Part 1: Removal of Variable wrapper in C++ https://github.com/pytorch/pytorch/pull/17072
• Part 2: Merge of Variable and Tensor types in C++ https://github.com/pytorch/pytorch/pull/28620

What's the current state of backend extensibility? How did PyTorch evolve from being a CPU and CUDA only framework to also support AMD ROCm and XLA? What are some problems with adding an out-of-tree backend, and what's some work to make it better?

Further reading:

• Script for HIPifying PyTorch's source when enabling ROCm https://github.com/pytorch/pytorch/blob/master/tools/amd_build/build_amd.py
• PyTorch/XLA https://github.com/pytorch/xla/
• Brian Hirsh's spec on what out-of-tree backend codegen looks like https://github.com/pytorch/xla/issues/2871

Structured kernels are a new way to write kernels in PyTorch. Why did they take so long? What finally convinced us that we should do them? Why did it end up taking me the better part of a year to only be half done with them?

Further reading:

• Structured kernels RFC https://github.com/pytorch/rfcs/blob/rfc-0005/RFC-0005-structured-kernel-definitions.md
• Taxonomy of PyTorch operators by shape behavior http://blog.ezyang.com/2020/05/a-brief-taxonomy-of-pytorch-operators-by-shape-behavior/
• Bram Wasti's lazy tensor prototype https://github.com/pytorch/pytorch/pull/25753

Functionalization is the process by which we remove mutation from autograd graphs in PyTorch, leaving us with a purely functional graph that we can execute in the normal way. Why do we need to do functionalization? What makes it not so easy to do? How do we do it? And how does it compare to mutation removal that you might see in a compiler?

Further reading:

• Section 3.1 of this paper on PyTorch AD https://openreview.net/pdf/25b8eee6c373d48b84e5e9c6e10e7cbbbce4ac73.pdf predates our implementation of inplace autograd but accurately reports the subtleties and correctly predicts the implementation strategy we ended up taking
• RFC to generalize the functionalization mechanism to be available to arbitrary backends https://github.com/pytorch/rfcs/pull/19
• Code that handles lazily updating views when the base is updated https://github.com/pytorch/pytorch/blob/e5e095cbe4dbc5a601f98e6134dcbd59c6342d7d/torch/csrc/autograd/variable.cpp#L556-L603

Ever wanted to learn about CUDA but not sure where to start? In this sixteen minute episode I try to jam in as much CUDA knowledge as could be reasonably expected in a podcast. You won't know how to write a kernel after this episode, but you'll know about what a GPU is, what the general CUDA programming model is, why asynchronous execution makes everything complicated, and some general principles PyTorch abides by when designing CUDA kernels.

Further reading:

• PyTorch docs on CUDA semantics https://pytorch.org/docs/stable/notes/cuda.html
• The book I was recommended for learning CUDA when I first showed up at PyToch: Programming Massively Parallel Processors https://www.amazon.com/Programming-Massively-Parallel-Processors-Hands/dp/0128119861
• The environment variable that makes CUDA synchronous is CUDA_LAUNCH_BLOCKING=1. cuda-memcheck is also useful for debugging CUDA problems https://docs.nvidia.com/cuda/cuda-memcheck/index.html

What's inference mode? Why doesn't my code run fast if I use no_grad or make sure requires_grad=False? How come inference mode is safe but AutoNonVariableTypeMode is not?

Further reading:

• Inference mode RFC https://github.com/.../rfc0011/RFC-0011-InferenceMode.md
• Inference mode docs for C++ frontend users https://github.com/.../cpp/source/notes/inference_mode.rst
• Tracking issue for Python frontend support https://github.com/pytorch/pytorch/issues/56608

What is vectorization? How do you use it in PyTorch? What are some of the traps and pitfalls of writing vectorized code in PyTorch?

Further reading:

• native/cpu README https://github.com/.../aten/src/ATen/native/cpu/README.md
• Vec256 classes https://github.com/.../tree/master/aten/src/ATen/cpu/vec256
• AVX512 support tracking issue https://github.com/pytorch/pytorch/issues/56187