Differentiable programming


Differentiable programming is a programming paradigm in which the programs can be differentiated throughout, usually via automatic differentiation. This allows for gradient based optimization of parameters in the program, often via gradient descent. Differentiable programming has found use in a wide variety of areas, particularly scientific computing and artificial intelligence.

Approaches

Most differentiable programming frameworks work by constructing a graph containing the control flow and data structures in the program. Earlier attempts generally fall into two groups:
Both of these early approaches are only able to differentiate code written in a suitable manner for the framework, limiting their interoperability with other programs.
More recent packages in the Julia programming language — , the Swift programming language — , and a new programming language — , resolve the issues that earlier attempts faced by treating the language's syntax as the graph. The intermediate representation of arbitrary code can then be differentiated directly, optimized, and compiled.

Applications

Differentiable programming has been applied in areas such as combining deep learning with physics engines in robotics, differentiable ray tracing, image processing, and probabilistic programming.