This document provides a comprehensive overview of differentiable programming, a paradigm enabling gradient-based optimization of computer programs, even those with complex control flows and data structures. It explores the fundamental concepts from automatic differentiation, including Jacobian and Hessian matrices, to the mathematical representation of programs as computation graphs and chains, encompassing neural network architectures like Transformers. The text further examines probabilistic learning methods, techniques for smoothing non-differentiable operations via optimization and integration, and various first and second-order optimization algorithms, highlighting the interplay between optimization, probability, and differentiation within this field.

Source: June 2025 - https://arxiv.org/pdf/2403.14606v3 - The Elements of

Differentiable Programming