PyTorch Developer Podcast

What's an expect test? Why should you use them? Why is inline expect test better than out of line? How to write a good expect test? 

Further reading. expecttest source implementation https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/expecttest.py (only 311 lines!)

What's PyTorch XLA? Why should you care? How is it implemented? How does PyTorch XLA trade off functionality versus ease of performance debugging? What are some new developments in this space?

Further reading.

• XLA's repo has lots of really good docs. Check out https://github.com/pytorch/xla/blob/master/OP_LOWERING_GUIDE.md and also the main https://github.com/pytorch/xla/blob/master/README.md
• Alex Suhan's RFC about lazy core https://github.com/pytorch/rfcs/pull/18

What is TH? Why might you care? What is so horrible about it? What the heck is the generic/ folder? Why are we porting everything to C++? What are some downsides of having ported all our TH code to C++?

Further reading. 

• The TH to ATen porting guide has lots of explanations of old school TH idioms https://github.com/pytorch/pytorch/wiki/TH-to-ATen-porting-guide
• Old notes about refcounting in TH https://github.com/pytorch/pytorch/blob/master/aten/src/README.md

There is a really good TorchScript overview at https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/OVERVIEW.md and in this 20min podcast, I want to give you some of the highlights from this document.

Why is PyTorch's build so g-dang complicated. How to avoid having to deal with cmake at all? And if you have to deal with cmake, what are the most important things to know? And if you were going to improve our cmake, how would you go about doing it...

Further reading.

• The official CMake documentation is a great help and well worth reading https://cmake.org/documentation
• If you work in torch/csrc chances are you'll need to edit this file https://github.com/pytorch/pytorch/blob/master/tools/build_variables.bzl

Liner notes.

• multiple build systems: cmake, buck, xplat buck, ovrsource buck, bazel
  • tools/build_variables.bzl is read from cmake! append_filelist
    • but not used uniformly for all components! (ouch!)
• mashed together ATen and Caffe2 build systems (e.g., main library libtorch_cpu is defined in caffe2/CMakeLists.txt)
• cmake: not very much syntax, "everything is a function". This means you can look up constructs relatively easily; e.g., even if() is a command
• the general cmake model: "set a bunch of variables, run a bunch of commands". cmake is VERY GREPPABLE
  • but not everything is in CMakeLists.txt; check *.cmake too
  • the directory structure makes no sense, you really need to grep.
    (doing a lot of set PARENT_SCOPE to propagate stuff)
  • renaming a file? grep for it
  • primary hazard of refactoring: need to make sure all the variables
    are setup at the new location
• many directories are not recursive glob, beware of adding new directories
• old school cmake: literally everything is stuffed in variables (CMAKE_CXX_FLAGS). new school cmake: attach things to targets, things propagate when you depend on targets (public/private dependencies)
• add_library: the most important thing
• don't randomly change things and pray: have hypotheses and test them

torchdeploy is a way of running multiple Python interpreters inside the same process. It can be used to deploy Python PyTorch programs in situations where the GIL is a problem, not the CPython interpreter. How does it work, and what kind of challenges does it pose for people who want to write code that calls from C++ to Python?

Further reading.

• How the torchdeploy build system works https://dev-discuss.pytorch.org/t/torch-deploy-the-build/238
• Description of the single interpreter per Tensor invariant https://github.com/pytorch/pytorch/issues/57756
• Recent work on making it possible to load C extensions into torchdeploy https://dev-discuss.pytorch.org/t/running-multiple-python-interpreters-via-custom-dynamic-loading/241

What's the C++ frontend? Why is avoiding templates so important? Why is Tensor a reference type? How do we simulate keyword arguments in C++? Where did the nn Module support in the C++ API come from? Why did we reimplement all modules in C++? How are modules implemented in C++? What are some performance challenges of writing Python in C++, and how are we working around them?

Further reading.

• C++ frontend tutorial https://pytorch.org/tutorials/advanced/cpp_frontend.html
• Writing Python in C++ (a manifesto) https://github.com/pytorch/pytorch/wiki/Writing-Python-in-cpp-(a-manifesto)
• MaybeOwned PR https://github.com/pytorch/pytorch/pull/53317

Given two separately refcounted objects, how can you arrange for each of them to stay live so long as the other is live? Why doesn't just having a strong-strong or strong-weak reference between the two objects work? What is object resurrection in CPython? What's a finalizer and why does it make things more complicated? How does Python GC work?

Further reading.

• PyObject preservation PR https://github.com/pytorch/pytorch/pull/56017
• Sam Gross's original PoC, which works fine if the two objects in question are both PyObjects https://github.com/colesbury/refcount/
• PEP 442 Safe object finalization https://www.python.org/dev/peps/pep-0442/
• Essential reading about Python GC https://devguide.python.org/garbage_collector/

What is mobile selective build? Why are we so obsessed with reducing binary size? How does selective build work? Why doesn't static linking just work? Why can't you just read out the ops used in a TorchScript model to determine what operators you actually need? What are the tradeoffs of statically determining the operator dependency graph versus tracing? What's up with the SELECTIVE_NAME macro? How the heck does selective build work at all when you have multiple mobile apps in a single Buck build system? What takeaways should I have as a regular PyTorch developer?

Further reading:

• Official open source mobile documentation on custom selective builds https://pytorch.org/mobile/android/#custom-build
• How to rebuild the op dependency yaml https://github.com/pytorch/pytorch/blob/master/tools/code_analyzer/build.sh

Liner notes:

• binary size is premium; ship only what you actually need

• big idea:

  • get the ops your model needs -> apply this to build of pytorch

• get the ops your model needs

  • TorchScript ~> read it out directly from the model itself
  • but what if ops use other ops?
    • need a dependency graph. done with static analysis llvm (jiakai) ~> with a (possibly inaccurate) yaml checked in for easy kickstart if you don't want to run the pass (updated by bot, not operational since Feb, recommend rebuilding from scratch if you run into trouble)
  • other possibility: dynamic tracing
    • pro: no need for dependency graph, just look at what was called; works for dtypes
    • con: need representative inputs, if control flow might not cover everything

• apply this to build of pytorch

  • ordinarily: static linking ensures stuff that isn't used gets pruned
    • but this doesn't work with distributed operator registration based on static initializers
  • how?
    • codegen - just don't generate it
    • no codegen - SELECTIVE_NAME - C++ doesn't support string in macro
  • build system integration
    • buck constraint: only one library
      • therefore: generate multiple copies of glue library
    • alt: atomize library into each operator. caffe2 used to do this; each library takes a long time to build (1m) and crashes xcode because there's too many

• common hiccups

  • modify implementation details, some op is/isn't called anymore ~> error! usually just means some yaml needs regenerating. PyTorch Edge developers are very friendly and can help

What goes into the implementation of torch.nn? Why do NN modules exist in the first place? What's the function of Parameter? How do modules actually track all the parameters in question? What is all of the goop in the top level NN module class? What are some new developments in torch.nn modules? What are some open problems with our modules?

Further reading:

• Implementation of nn.Module https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py
• nn.Module is complicated and that means its sometimes a bit slow. Some analysis at https://dev-discuss.pytorch.org/t/overhead-in-nn-module-causing-massive-slowdowns-compared-to-raw-cublas-or-torchscript/110
• Lazy modules PR https://github.com/pytorch/pytorch/pull/44538 and factory kwargs https://github.com/pytorch/pytorch/pull/54508

Liner notes:

• python for hackability (c++ is reimplemented)
• parameters
  • parameter collection (for optimization)
  • buffers: not considered optimizable
• modules
  • functorial operation (_apply)
  • jit script: staged computation (init is not scripted)
  • __call__ to forward (extra instrumentation)
  • serialization / state_dict
• new stuff: device kwarg (joel schlosser)
• new stuff: lazy modules (emcastillo)
• open problems: parameter initialization