diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index 9ae20f753..543267785 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.10.2","generation_timestamp":"2024-03-24T16:23:43","documenter_version":"1.3.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.10.2","generation_timestamp":"2024-03-24T19:53:09","documenter_version":"1.3.0"}} \ No newline at end of file diff --git a/dev/api/index.html b/dev/api/index.html index 13d6c63ea..6869e9c21 100644 --- a/dev/api/index.html +++ b/dev/api/index.html @@ -1,17 +1,17 @@ -API reference · DifferentiationInterface.jl
GitHub

API reference

DifferentiationInterface.DifferentiationInterface — Module
source

Derivative

Gradient

Jacobian

Second order

DifferentiationInterface.SecondOrder — Type
SecondOrder

Combination of two backends for second-order differentiation.

Fields

  • outer::ADTypes.AbstractADType: backend for the outer differentiation

  • inner::ADTypes.AbstractADType: backend for the inner differentiation

source

Primitives

Backend queries

Preparation

Testing & benchmarking

DifferentiationInterface.DifferentiationTest.BenchmarkData — Type
BenchmarkData

Ad-hoc storage type for differentiation benchmarking results. You can turn it into a DataFrame as follows:

df = DataFrames.DataFrame(pairs(benchmark_data)...)

Fields

These are not part of the public API.

  • backend::Vector{String}

  • mode::Vector{Type}

  • operator::Vector{Function}

  • variant::Vector{Function}

  • func::Vector{String}

  • mutating::Vector{Bool}

  • input_type::Vector{Type}

  • output_type::Vector{Type}

  • input_size::Vector

  • output_size::Vector

  • samples::Vector{Int64}

  • time::Vector{Float64}

  • bytes::Vector{Float64}

  • allocs::Vector{Float64}

  • compile_fraction::Vector{Float64}

  • gc_fraction::Vector{Float64}

  • evals::Vector{Float64}

source
DifferentiationInterface.DifferentiationTest.test_differentiation — Function
test_differentiation(backends, [operators, scenarios]; [kwargs...])

Cross-test a list of backends for a list of operators on a list of scenarios, running a variety of different tests.

  • If benchmark is false, this returns a TestSet object.
  • If benchmark is true, this returns a BenchmarkData object, which is easy to turn into a DataFrame.

Default arguments

  • operators::Vector{Function}: the list [pushforward, pullback,derivative, gradient, jacobian, second_derivative, hvp, hessian]
  • scenarios::Vector{Scenario}: the output of default_scenarios()

Keyword arguments

Testing:

  • correctness=true: whether to compare the differentiation results with those given by ForwardDiff.jl
  • error_free=false: whether to run it once and see if it errors
  • type_stability=false: whether to check type stability with JET.jl (@test_opt)
  • call_count=false: whether to check that the function is called the right number of times
  • benchmark=false: whether to run and return a benchmark suite with Chairmarks.jl
  • allocations=false: whether to check that the benchmarks are allocation-free
  • detailed=false: whether to print a detailed test set (by scenario) or condensed test set (by operator)

Filtering:

  • input_type=Any: restrict scenario inputs to subtypes of this
  • output_type=Any: restrict scenario outputs to subtypes of this
  • first_order=true: consider first order operators
  • second_order=true: consider second order operators
  • allocating=true: consider operators for allocating functions
  • mutating=true: consider operators for mutating functions
  • excluded=Symbol[]: list of excluded operators
source
DifferentiationInterface.DifferentiationTest.test_differentiation — Method
test_differentiation(
+API reference · DifferentiationInterface.jl
GitHub
API reference
DifferentiationInterface.DifferentiationInterface — Module
source
Derivative
Gradient
Jacobian
Second order
DifferentiationInterface.SecondOrder — Type
SecondOrder
Combination of two backends for second-order differentiation.
Fields
  • outer::ADTypes.AbstractADType: backend for the outer differentiation
  • inner::ADTypes.AbstractADType: backend for the inner differentiation
source
Primitives
Backend queries
Preparation
Testing & benchmarking
DifferentiationInterface.DifferentiationTest.BenchmarkData — Type
BenchmarkData
Ad-hoc storage type for differentiation benchmarking results. You can turn it into a DataFrame as follows:
df = DataFrames.DataFrame(pairs(benchmark_data)...)
Fields
These are not part of the public API.
  • backend::Vector{String}
  • mode::Vector{Type}
  • operator::Vector{Function}
  • variant::Vector{Function}
  • func::Vector{String}
  • mutating::Vector{Bool}
  • input_type::Vector{Type}
  • output_type::Vector{Type}
  • input_size::Vector
  • output_size::Vector
  • samples::Vector{Int64}
  • time::Vector{Float64}
  • bytes::Vector{Float64}
  • allocs::Vector{Float64}
  • compile_fraction::Vector{Float64}
  • gc_fraction::Vector{Float64}
  • evals::Vector{Float64}
source
DifferentiationInterface.DifferentiationTest.test_differentiation — Function
test_differentiation(backends, [operators, scenarios]; [kwargs...])
Cross-test a list of backends for a list of operators on a list of scenarios, running a variety of different tests.
  • If benchmark is false, this returns a TestSet object.
  • If benchmark is true, this returns a BenchmarkData object, which is easy to turn into a DataFrame.
Default arguments
  • operators::Vector{Function}: the list [pushforward, pullback,derivative, gradient, jacobian, second_derivative, hvp, hessian]
  • scenarios::Vector{Scenario}: the output of default_scenarios()
Keyword arguments
Testing:
  • correctness=true: whether to compare the differentiation results with those given by ForwardDiff.jl
  • error_free=false: whether to run it once and see if it errors
  • call_count=false: whether to check that the function is called the right number of times
  • type_stability=false: whether to check type stability with JET.jl (@test_opt)
  • benchmark=false: whether to run and return a benchmark suite with Chairmarks.jl
  • allocations=false: whether to check that the benchmarks are allocation-free
  • detailed=false: whether to print a detailed test set (by scenario) or condensed test set (by operator)
Filtering:
  • input_type=Any: restrict scenario inputs to subtypes of this
  • output_type=Any: restrict scenario outputs to subtypes of this
  • first_order=true: consider first order operators
  • second_order=true: consider second order operators
  • allocating=true: consider operators for allocating functions
  • mutating=true: consider operators for mutating functions
  • excluded=Symbol[]: list of excluded operators
source
Internals
This is not part of the public API.
DifferentiationInterface.basisarray — Method
basisarray(backend, a::AbstractArray, i::CartesianIndex)
Construct the i-th stardard basis array in the vector space of a with element type eltype(a).
Note
If an AD backend benefits from a more specialized basis array implementation, this function can be extended on the backend type.
source
DifferentiationInterface.mode — Method
mode(backend)
Return the AD mode of a backend in a statically predictable way.
The return value is a Type object chosen among:
  • ADTypes.AbstractForwardMode
  • ADTypes.AbstractFiniteDifferencesMode
  • ADTypes.AbstractReverseMode
  • ADTypes.AbstractSymbolicDifferentiationMode
This function exists because there are backends (like Enzyme) that can support both forward and reverse mode, which means their ADTypes.jl object does not subtype either.
source

+
diff --git a/dev/backends/index.html b/dev/backends/index.html index 7d972d16b..638482764 100644 --- a/dev/backends/index.html +++ b/dev/backends/index.html @@ -1,10 +1,10 @@ Backends · DifferentiationInterface.jl
GitHub

Backends

Types

Most backend choices are defined by ADTypes.jl.

Warning

Only the backends listed here are supported by DifferentiationInterface.jl, even though ADTypes.jl defines more.

ADTypes.AutoEnzyme — Type
AutoEnzyme(Enzyme.Forward)
-AutoEnzyme(Enzyme.Reverse)

Construct a forward or reverse mode AutoEnzyme backend.

source
AutoEnzyme{M}

Chooses Enzyme.jl.

Fields

  • mode::M = nothing
source

We also provide a few of our own:

Availability

You can use check_available to verify whether a given backend is loaded, like we did below:

Backendavailable
Diffractor (forward)✓
Enzyme (forward)✓
Enzyme (reverse)✓
FastDifferentiation (symbolic)✓
FiniteDiff (finite)✓
FiniteDifferences (finite)✓
ForwardDiff (forward)✓
PolyesterForwardDiff (forward)✓
ReverseDiff (reverse)✗
Tracker (reverse)✓
Zygote (reverse)✓
+AutoEnzyme(Enzyme.Reverse)

Construct a forward or reverse mode AutoEnzyme backend.

source
AutoEnzyme{M}

Chooses Enzyme.jl.

Fields

  • mode::M = nothing
source

We also provide a few of our own:

Availability

You can use check_available to verify whether a given backend is loaded, like we did below:

Backendavailable
Diffractor (forward)✓
Enzyme (forward)✓
Enzyme (reverse)✓
FastDifferentiation (symbolic)✓
FiniteDiff (finite)✓
FiniteDifferences (finite)✓
ForwardDiff (forward)✓
PolyesterForwardDiff (forward)✓
ReverseDiff (reverse)✗
Tracker (reverse)✓
Zygote (reverse)✓

Mutation support

All backends are compatible with allocating functions f(x) = y. Only some are compatible with mutating functions f!(y, x) = nothing. You can use check_mutation to check that feature, like we did below:

Backendmutation
Diffractor (forward)✗
Enzyme (forward)✓
Enzyme (reverse)✓
FastDifferentiation (symbolic)✗
FiniteDiff (finite)✓
FiniteDifferences (finite)✗
ForwardDiff (forward)✓
PolyesterForwardDiff (forward)✓
ReverseDiff (reverse)✓
Tracker (reverse)✗
Zygote (reverse)✗

Package extensions

Backend-specific extension content is not part of the public API.

+
diff --git a/dev/developer/index.html b/dev/developer/index.html index 50511b912..218370fce 100644 --- a/dev/developer/index.html +++ b/dev/developer/index.html @@ -18,4 +18,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/dev/index.html b/dev/index.html index 0a3624f49..0d136166d 100644 --- a/dev/index.html +++ b/dev/index.html @@ -13,4 +13,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/dev/objects.inv b/dev/objects.inv index 5c7e9ef68..a6911d737 100644 Binary files a/dev/objects.inv and b/dev/objects.inv differ diff --git a/dev/overview/index.html b/dev/overview/index.html index 087ad555e..e387675ed 100644 --- a/dev/overview/index.html +++ b/dev/overview/index.html @@ -1,8 +1,8 @@ -Overview · DifferentiationInterface.jl
GitHub

Overview

Operators

Depending on the type of input and output, differentiation operators can have various names. Most backends have custom implementations, which we reuse if possible.

We choose the following terminology for the high-level operators we provide:

operatorinput xoutput yresult typeresult shape
derivativeNumberAnysame as ysize(y)
gradientAnyNumbersame as xsize(x)
jacobianAbstractArrayAbstractArrayAbstractMatrix(length(y), length(x))

They are all based on the following low-level operators:

  • pushforward (or JVP), to propagate input tangents
  • pullback (or VJP), to backpropagate output cotangents
Tip

See the book The Elements of Differentiable Programming for details on these concepts.

Variants

Several variants of each operator are defined:

out-of-placein-place (or not)out-of-place + primalin-place (or not) + primal
derivativederivative!!value_and_derivativevalue_and_derivative!!
gradientgradient!!value_and_gradientvalue_and_gradient!!
jacobianjacobian!!value_and_jacobianvalue_and_jacobian!!
pushforwardpushforward!!value_and_pushforwardvalue_and_pushforward!!
pullbackpullback!!value_and_pullbackvalue_and_pullback!!
Warning

The "bang-bang" syntactic convention !! signals that some of the arguments can be mutated, but they do not have to be. Users should not rely on mutation, but instead recover the function output and work from there.

grad = gradient!!(f, grad, backend, x)  # good
+Overview · DifferentiationInterface.jl
GitHub
Overview
Operators
Depending on the type of input and output, differentiation operators can have various names. Most backends have custom implementations, which we reuse if possible.
We choose the following terminology for the high-level operators we provide:
operatorinput  xoutput   yresult typeresult shape
derivativeNumberAnysame as ysize(y)
gradientAnyNumbersame as xsize(x)
jacobianAbstractArrayAbstractArrayAbstractMatrix(length(y), length(x))
They are all based on the following low-level operators:
  • pushforward (or JVP), to propagate input tangents
  • pullback (or VJP), to backpropagate output cotangents
Tip
See the book The Elements of Differentiable Programming for details on these concepts.
Variants
Several variants of each operator are defined:
out-of-placein-place (or not)out-of-place + primalin-place (or not) + primal
derivativederivative!!value_and_derivativevalue_and_derivative!!
gradientgradient!!value_and_gradientvalue_and_gradient!!
jacobianjacobian!!value_and_jacobianvalue_and_jacobian!!
pushforwardpushforward!!value_and_pushforwardvalue_and_pushforward!!
pullbackpullback!!value_and_pullbackvalue_and_pullback!!
Warning
The "bang-bang" syntactic convention !! signals that some of the arguments can be mutated, but they do not have to be. Users should not rely on mutation, but instead recover the function output and work from there.
grad = gradient!!(f, grad, backend, x)  # good
 gradient!!(f, grad, backend, x)  # bad, cause `grad` may not have changed
Second order
Second-order differentiation is also supported, with the following operators:
operatorinput  xoutput   yresult typeresult shape
second_derivativeNumberAnysame as ysize(y)
hvpAnyNumbersame as xsize(x)
hessianAbstractArrayNumberAbstractMatrix(length(x), length(x))
Danger
This is an experimental functionality, use at your own risk.
Preparation
In many cases, automatic differentiation can be accelerated if the function has been run at least once (e.g. to record a tape) and if some cache objects are provided. This is a backend-specific procedure, but we expose a common syntax to achieve it.
operatorpreparation function
derivativeprepare_derivative
gradientprepare_gradient
jacobianprepare_jacobian
second_derivativeprepare_second_derivative
hessianprepare_hessian
pushforwardprepare_pushforward
pullbackprepare_pullback
hvpprepare_hvp
If you run prepare_operator(backend, f, x), it will create an object called extras containing the necessary information to speed up operator and its variants. This information is specific to backend and f, as well as the type and size of the input x, but it should work with different values of x.
You can then call operator(backend, f, similar_x, extras), which should be faster than operator(backend, f, similar_x). This is especially worth it if you plan to call operator several times in similar settings: you can think of it as a warm up.
By default, all the preparation functions return nothing. We do not make any guarantees on their implementation for each backend, or on the performance gains that can be expected.
Warning
We haven't fully figured out what must happen when an extras object is prepared for a specific operator but then given to a lower-level one (i.e. prepare it for jacobian but then give it to pushforward inside jacobian).
Multiple inputs/outputs
Restricting the API to one input and one output has many coding advantages, but it is not very flexible. If you need more than that, use ComponentArrays.jl to wrap several objects inside a single ComponentVector.

+
diff --git a/dev/search_index.js b/dev/search_index.js index 8c71295f0..7f53564a8 100644 --- a/dev/search_index.js +++ b/dev/search_index.js @@ -1,3 +1,3 @@ var documenterSearchIndex = {"docs": -[{"location":"api/","page":"API reference","title":"API reference","text":"CurrentModule = DifferentiationInterface\nCollapsedDocStrings = true","category":"page"},{"location":"api/#API-reference","page":"API reference","title":"API reference","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"DifferentiationInterface","category":"page"},{"location":"api/#DifferentiationInterface.DifferentiationInterface","page":"API reference","title":"DifferentiationInterface.DifferentiationInterface","text":"DifferentiationInterface\n\nAn interface to various automatic differentiation backends in Julia.\n\nExports\n\nAutoFastDifferentiation\nSecondOrder\ncheck_available\ncheck_hessian\ncheck_mutation\nderivative\nderivative!!\ngradient\ngradient!!\nhessian\nhvp\njacobian\njacobian!!\nprepare_derivative\nprepare_gradient\nprepare_hessian\nprepare_hvp\nprepare_jacobian\nprepare_pullback\nprepare_pushforward\nprepare_second_derivative\npullback\npullback!!\npushforward\npushforward!!\nsecond_derivative\nvalue_and_derivative\nvalue_and_derivative!!\nvalue_and_gradient\nvalue_and_gradient!!\nvalue_and_jacobian\nvalue_and_jacobian!!\nvalue_and_pullback\nvalue_and_pullback!!\nvalue_and_pushforward\nvalue_and_pushforward!!\n\n\n\n\n\n","category":"module"},{"location":"api/#Derivative","page":"API reference","title":"Derivative","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"src/derivative.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.derivative!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Number}, Tuple{F, Any, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.derivative!!","text":"derivative!!(f, der, backend, x, [extras]) -> der\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.derivative-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Number}, Tuple{F, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.derivative","text":"derivative(f, backend, x, [extras]) -> der\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_derivative!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Number}, Tuple{F, Any, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_derivative!!","text":"value_and_derivative!!(f, der, backend, x, [extras]) -> (y, der)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_derivative!!-Union{Tuple{F}, Tuple{F, Any, Any, ADTypes.AbstractADType, Number}, Tuple{F, Any, Any, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_derivative!!","text":"value_and_derivative!!(f!, y, der, backend, x, [extras]) -> (y, der)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_derivative-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Number}, Tuple{F, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_derivative","text":"value_and_derivative(f, backend, x, [extras]) -> (y, der)\n\n\n\n\n\n","category":"method"},{"location":"api/#Gradient","page":"API reference","title":"Gradient","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"gradient.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.gradient!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.gradient!!","text":"gradient!!(f, grad, backend, x, [extras]) -> grad\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.gradient-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.gradient","text":"gradient(f, backend, x, [extras]) -> grad\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_gradient!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_gradient!!","text":"value_and_gradient!!(f, grad, backend, x, [extras]) -> (y, grad)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_gradient-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_gradient","text":"value_and_gradient(f, backend, x, [extras]) -> (y, grad)\n\n\n\n\n\n","category":"method"},{"location":"api/#Jacobian","page":"API reference","title":"Jacobian","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"jacobian.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.jacobian!!-Union{Tuple{F}, Tuple{F, AbstractMatrix, ADTypes.AbstractADType, AbstractArray}, Tuple{F, AbstractMatrix, ADTypes.AbstractADType, AbstractArray, Any}} where F","page":"API reference","title":"DifferentiationInterface.jacobian!!","text":"jacobian!!(f, jac, backend, x, [extras]) -> jac\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.jacobian-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, AbstractArray}, Tuple{F, ADTypes.AbstractADType, AbstractArray, Any}} where F","page":"API reference","title":"DifferentiationInterface.jacobian","text":"jacobian(f, backend, x, [extras]) -> jac\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_jacobian!!-Union{Tuple{F}, Tuple{F, AbstractArray, AbstractMatrix, ADTypes.AbstractADType, AbstractArray}, Tuple{F, AbstractArray, AbstractMatrix, ADTypes.AbstractADType, AbstractArray, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_jacobian!!","text":"value_and_jacobian!!(f!, y, jac, backend, x, [extras]) -> (y, jac)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_jacobian!!-Union{Tuple{F}, Tuple{F, AbstractMatrix, ADTypes.AbstractADType, AbstractArray}, Tuple{F, AbstractMatrix, ADTypes.AbstractADType, AbstractArray, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_jacobian!!","text":"value_and_jacobian!!(f, jac, backend, x, [extras]) -> (y, jac)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_jacobian-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, AbstractArray}, Tuple{F, ADTypes.AbstractADType, AbstractArray, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_jacobian","text":"value_and_jacobian(f, backend, x, [extras]) -> (y, jac)\n\n\n\n\n\n","category":"method"},{"location":"api/#Second-order","page":"API reference","title":"Second order","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"second_order.jl\", \"second_derivative.jl\", \"hessian.jl\", \"hvp.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.SecondOrder","page":"API reference","title":"DifferentiationInterface.SecondOrder","text":"SecondOrder\n\nCombination of two backends for second-order differentiation.\n\nFields\n\nouter::ADTypes.AbstractADType: backend for the outer differentiation\ninner::ADTypes.AbstractADType: backend for the inner differentiation\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.second_derivative-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Number}, Tuple{F, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.second_derivative","text":"second_derivative(f, backend, x, [extras]) -> der2\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.hessian-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.hessian","text":"hessian(f, backend, x, [extras]) -> hess\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.hvp-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.hvp","text":"hvp(f, backend, x, v, [extras]) -> p\n\n\n\n\n\n","category":"method"},{"location":"api/#Primitives","page":"API reference","title":"Primitives","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"pushforward.jl\", \"pullback.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.pushforward!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.pushforward!!","text":"pushforward!!(f, dy, backend, x, dx, [extras]) -> (y, dy)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.pushforward-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.pushforward","text":"pushforward(f, backend, x, dx, [extras]) -> (y, dy)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pushforward!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pushforward!!","text":"value_and_pushforward!!(f, dy, backend, x, dx, [extras]) -> (y, dy)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pushforward!!-Union{Tuple{F}, Tuple{F, Any, Any, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pushforward!!","text":"value_and_pushforward!!(f!, y, dy, backend, x, dx, [extras]) -> (y, dy)\n\ninfo: Info\nRequired primitive for forward mode backends to support mutating functions.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pushforward-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pushforward","text":"value_and_pushforward(f, backend, x, dx, [extras]) -> (y, dy)\n\ninfo: Info\nRequired primitive for forward mode backends to support allocating functions.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.pullback!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.pullback!!","text":"pullback!!(f, dx, backend, x, dy, [extras]) -> dx\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.pullback-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.pullback","text":"pullback(f, backend, x, dy, [extras]) -> dx\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pullback!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pullback!!","text":"value_and_pullback!!(f, dx, backend, x, dy, [extras]) -> (y, dx)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pullback!!-Union{Tuple{F}, Tuple{F, Any, Any, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pullback!!","text":"value_and_pullback!!(f!, y, dx, backend, x, dy, [extras]) -> (y, dx)\n\ninfo: Info\nRequired primitive for reverse mode backends to support mutating functions.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pullback-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pullback","text":"value_and_pullback(f, backend, x, dy, [extras]) -> (y, dx)\n\ninfo: Info\nRequired primitive for reverse mode backends to support allocating functions.\n\n\n\n\n\n","category":"method"},{"location":"api/#Backend-queries","page":"API reference","title":"Backend queries","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"backends.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.check_available-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.check_available","text":"check_available(backend)\n\nCheck whether backend is available by trying a scalar-to-scalar derivative.\n\nwarning: Warning\nMight take a while due to compilation time.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.check_hessian-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.check_hessian","text":"check_hessian(backend)\n\nCheck whether backend supports second order differentiation by trying a hessian.\n\nwarning: Warning\nMight take a while due to compilation time.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.check_mutation-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.check_mutation","text":"check_mutation(backend)\n\nCheck whether backend supports differentiation of mutating functions by trying a jacobian.\n\nwarning: Warning\nMight take a while due to compilation time.\n\n\n\n\n\n","category":"method"},{"location":"api/#Preparation","page":"API reference","title":"Preparation","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"prepare.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.prepare_derivative-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_derivative","text":"prepare_derivative(f, backend, x) -> extras\nprepare_derivative(f!, backend, y, x) -> extras\n\nCreate an extras object that can be given to derivative operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_gradient-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_gradient","text":"prepare_gradient(f, backend, x) -> extras\n\nCreate an extras object that can be given to gradient operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_hessian-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_hessian","text":"prepare_hessian(f, backend, x) -> extras\n\nCreate an extras object that can be given to Hessian operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_hvp-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_hvp","text":"prepare_hvp(f, backend, x) -> extras\n\nCreate an extras object that can be given to Hessian-vector product operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_jacobian-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_jacobian","text":"prepare_jacobian(f, backend, x) -> extras\nprepare_jacobian(f!, backend, y, x) -> extras\n\nCreate an extras object that can be given to Jacobian operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_pullback-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_pullback","text":"prepare_pullback(f, backend, x) -> extras\nprepare_pullback(f!, backend, y, x) -> extras\n\nCreate an extras object that can be given to pullback operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_pushforward-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_pushforward","text":"prepare_pushforward(f, backend, x) -> extras\nprepare_pushforward(f!, backend, y, x) -> extras\n\nCreate an extras object that can be given to pushforward operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_second_derivative-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_second_derivative","text":"prepare_second_derivative(f, backend, x) -> extras\nprepare_second_derivative(f!, backend, y, x) -> extras\n\nCreate an extras object that can be given to second derivative operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#Testing-and-benchmarking","page":"API reference","title":"Testing & benchmarking","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationTest]\nPrivate = false","category":"page"},{"location":"api/#DifferentiationInterface.DifferentiationTest","page":"API reference","title":"DifferentiationInterface.DifferentiationTest","text":"DifferentiationInterface.DifferentiationTest\n\nTesting utilities for DifferentiationInterface.\n\n\n\n\n\n","category":"module"},{"location":"api/#DifferentiationInterface.DifferentiationTest.BenchmarkData","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.BenchmarkData","text":"BenchmarkData\n\nAd-hoc storage type for differentiation benchmarking results. You can turn it into a DataFrame as follows:\n\ndf = DataFrames.DataFrame(pairs(benchmark_data)...)\n\nFields\n\nThese are not part of the public API.\n\nbackend::Vector{String}\nmode::Vector{Type}\noperator::Vector{Function}\nvariant::Vector{Function}\nfunc::Vector{String}\nmutating::Vector{Bool}\ninput_type::Vector{Type}\noutput_type::Vector{Type}\ninput_size::Vector\noutput_size::Vector\nsamples::Vector{Int64}\ntime::Vector{Float64}\nbytes::Vector{Float64}\nallocs::Vector{Float64}\ncompile_fraction::Vector{Float64}\ngc_fraction::Vector{Float64}\nevals::Vector{Float64}\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.DifferentiationTest.Scenario","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.Scenario","text":"Scenario{mutating}\n\nStore a testing scenario composed of a function and its input + output + tangents.\n\nFields\n\nf::Any: function\nx::Any: input\ny::Any: output\ndx::Any: pushforward seed\ndy::Any: pullback seed\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.DifferentiationTest.all_operators-Tuple{}","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.all_operators","text":"all_operators()\n\nList all operators that can be tested with test_differentiation.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.DifferentiationTest.default_scenarios-Tuple{}","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.default_scenarios","text":"default_scenarios()\n\nCreate a vector of Scenarios for testing differentiation. \n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.DifferentiationTest.test_differentiation","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.test_differentiation","text":"test_differentiation(backends, [operators, scenarios]; [kwargs...])\n\nCross-test a list of backends for a list of operators on a list of scenarios, running a variety of different tests.\n\nIf benchmark is false, this returns a TestSet object.\nIf benchmark is true, this returns a BenchmarkData object, which is easy to turn into a DataFrame.\n\nDefault arguments\n\noperators::Vector{Function}: the list [pushforward, pullback,derivative, gradient, jacobian, second_derivative, hvp, hessian]\nscenarios::Vector{Scenario}: the output of default_scenarios()\n\nKeyword arguments\n\nTesting:\n\ncorrectness=true: whether to compare the differentiation results with those given by ForwardDiff.jl\nerror_free=false: whether to run it once and see if it errors\ntype_stability=false: whether to check type stability with JET.jl (@test_opt)\ncall_count=false: whether to check that the function is called the right number of times\nbenchmark=false: whether to run and return a benchmark suite with Chairmarks.jl\nallocations=false: whether to check that the benchmarks are allocation-free\ndetailed=false: whether to print a detailed test set (by scenario) or condensed test set (by operator)\n\nFiltering:\n\ninput_type=Any: restrict scenario inputs to subtypes of this\noutput_type=Any: restrict scenario outputs to subtypes of this\nfirst_order=true: consider first order operators\nsecond_order=true: consider second order operators\nallocating=true: consider operators for allocating functions\nmutating=true: consider operators for mutating functions\nexcluded=Symbol[]: list of excluded operators\n\n\n\n\n\n","category":"function"},{"location":"api/#DifferentiationInterface.DifferentiationTest.test_differentiation-Tuple{ADTypes.AbstractADType, Vararg{Any}}","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.test_differentiation","text":"test_differentiation(\n backend::ADTypes.AbstractADType,\n args...;\n kwargs...\n) -> Union{Test.FallbackTestSet, BenchmarkData, Test.DefaultTestSet}\n\n\nShortcut for a single backend.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.DifferentiationTest.weird_array_scenarios-Tuple{}","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.weird_array_scenarios","text":"weird_array_scenarios()\n\nCreate a vector of Scenarios involving weird types for testing differentiation.\n\n\n\n\n\n","category":"method"},{"location":"api/#Internals","page":"API reference","title":"Internals","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"This is not part of the public API.","category":"page"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPublic = false\nOrder = [:function, :type]\nFilter = t -> !(t isa Type && t <: ADTypes.AbstractADType)","category":"page"},{"location":"api/#DifferentiationInterface.basisarray-Tuple{ADTypes.AbstractADType, AbstractArray, Any}","page":"API reference","title":"DifferentiationInterface.basisarray","text":"basisarray(backend, a::AbstractArray, i::CartesianIndex)\n\nConstruct the i-th stardard basis array in the vector space of a with element type eltype(a).\n\nNote\n\nIf an AD backend benefits from a more specialized basis array implementation, this function can be extended on the backend type.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.mode-Tuple{ADTypes.AbstractForwardMode}","page":"API reference","title":"DifferentiationInterface.mode","text":"mode(backend)\n\nReturn the AD mode of a backend in a statically predictable way.\n\nThe return value is a Type object chosen among:\n\nADTypes.AbstractForwardMode\nADTypes.AbstractFiniteDifferencesMode\nADTypes.AbstractReverseMode\nADTypes.AbstractSymbolicDifferentiationMode\n\nThis function exists because there are backends (like Enzyme) that can support both forward and reverse mode, which means their ADTypes.jl object does not subtype either.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.supports_mutation-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.supports_mutation","text":"supports_mutation(backend)\n\nReturn MutationSupported or MutationNotSupported in a statically predictable way.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.supports_pullback-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.supports_pullback","text":"supports_pullback(backend)\n\nReturn PullbackSupported or PullbackNotSupported in a statically predictable way.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.supports_pushforward-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.supports_pushforward","text":"supports_pushforward(backend)\n\nReturn PushforwardSupported or PushforwardNotSupported in a statically predictable way.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.ForwardOverForward","page":"API reference","title":"DifferentiationInterface.ForwardOverForward","text":"ForwardOverForward\n\nTraits identifying second-order backends that compute HVPs in forward over forward mode (inefficient).\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.ForwardOverReverse","page":"API reference","title":"DifferentiationInterface.ForwardOverReverse","text":"ForwardOverReverse\n\nTraits identifying second-order backends that compute HVPs in forward over reverse mode.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.MutationNotSupported","page":"API reference","title":"DifferentiationInterface.MutationNotSupported","text":"MutationNotSupported\n\nTrait identifying backends that do not support mutating functions f!(y, x).\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.MutationSupported","page":"API reference","title":"DifferentiationInterface.MutationSupported","text":"MutationSupported\n\nTrait identifying backends that support mutating functions f!(y, x).\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.PullbackNotSupported","page":"API reference","title":"DifferentiationInterface.PullbackNotSupported","text":"PullbackNotSupported\n\nTrait identifying backends that do not support efficient pullbacks.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.PullbackSupported","page":"API reference","title":"DifferentiationInterface.PullbackSupported","text":"PullbackSupported\n\nTrait identifying backends that support efficient pullbacks.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.PushforwardNotSupported","page":"API reference","title":"DifferentiationInterface.PushforwardNotSupported","text":"PushforwardNotSupported\n\nTrait identifying backends that do not support efficient pushforwards.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.PushforwardSupported","page":"API reference","title":"DifferentiationInterface.PushforwardSupported","text":"PushforwardSupported\n\nTrait identifying backends that support efficient pushforwards.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.ReverseOverForward","page":"API reference","title":"DifferentiationInterface.ReverseOverForward","text":"ReverseOverForward\n\nTraits identifying second-order backends that compute HVPs in reverse over forward mode.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.ReverseOverReverse","page":"API reference","title":"DifferentiationInterface.ReverseOverReverse","text":"ReverseOverReverse\n\nTraits identifying second-order backends that compute HVPs in reverse over reverse mode.\n\n\n\n\n\n","category":"type"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationTest]\nPublic = false","category":"page"},{"location":"api/#DifferentiationInterface.DifferentiationTest.AutoZeroForward","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.AutoZeroForward","text":"AutoZeroForward <: ADTypes.AbstractForwardMode\n\nTrivial backend that sets all derivatives to zero. Used in testing and benchmarking.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.DifferentiationTest.AutoZeroReverse","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.AutoZeroReverse","text":"AutoZeroReverse <: ADTypes.AbstractReverseMode\n\nTrivial backend that sets all derivatives to zero. Used in testing and benchmarking.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.DifferentiationTest.backend_string-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.backend_string","text":"backend_string(backend)\n\nReturn a shorter string than the full object printing from ADTypes.jl. Might be ambiguous.\n\n\n\n\n\n","category":"method"},{"location":"api/","page":"API reference","title":"API reference","text":"","category":"page"},{"location":"developer/#For-AD-developers","page":"For AD developers","title":"For AD developers","text":"","category":"section"},{"location":"developer/#Backend-requirements","page":"For AD developers","title":"Backend requirements","text":"","category":"section"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"To be usable with DifferentiationInterface.jl, an AD backend needs an object subtyping ADTypes.AbstractADType. In addition, some operators must be defined:","category":"page"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"backend subtype pushforward necessary pullback necessary\nADTypes.AbstractForwardMode yes no\nADTypes.AbstractFiniteDifferencesMode yes no\nADTypes.AbstractReverseMode no yes\nADTypes.AbstractSymbolicDifferentiationMode yes yes","category":"page"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"Every backend we support corresponds to a package extension of DifferentiationInterface.jl (located in the ext subfolder). Advanced users are welcome to code more backends and submit pull requests!","category":"page"},{"location":"developer/#Fallback-call-structure","page":"For AD developers","title":"Fallback call structure","text":"","category":"section"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"For simplicity, we remove value_ in the operator names below.","category":"page"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"note: Edge labels\nFull edges in the following graphs require a single call to the destination. Dotted edges require multiple calls to the destination, the number is indicated above.","category":"page"},{"location":"developer/#Forward-mode,-allocating-functions","page":"For AD developers","title":"Forward mode, allocating functions","text":"","category":"section"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"flowchart LR\n pushforward!! --> pushforward\n derivative --> pushforward\n derivative!! --> pushforward!!\n gradient .-> |n|pushforward\n gradient!! .-> |n|pushforward!!\n jacobian .-> |n|pushforward\n jacobian!! .-> |n|pushforward!!","category":"page"},{"location":"developer/#Reverse-mode,-allocating-functions","page":"For AD developers","title":"Reverse mode, allocating functions","text":"","category":"section"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"flowchart LR\n pullback!! --> pullback\n derivative .-> |m|pullback\n derivative!! .-> |m|pullback!!\n gradient --> pullback\n gradient!! --> pullback!!\n jacobian .-> |m|pullback\n jacobian!! .-> |m|pullback!!","category":"page"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"","category":"page"},{"location":"overview/#Overview","page":"Overview","title":"Overview","text":"","category":"section"},{"location":"overview/#operators","page":"Overview","title":"Operators","text":"","category":"section"},{"location":"overview/","page":"Overview","title":"Overview","text":"Depending on the type of input and output, differentiation operators can have various names. Most backends have custom implementations, which we reuse if possible.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"We choose the following terminology for the high-level operators we provide:","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"operator input x output y result type result shape\nderivative Number Any same as y size(y)\ngradient Any Number same as x size(x)\njacobian AbstractArray AbstractArray AbstractMatrix (length(y), length(x))","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"They are all based on the following low-level operators:","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"pushforward (or JVP), to propagate input tangents\npullback (or VJP), to backpropagate output cotangents","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"tip: Tip\nSee the book The Elements of Differentiable Programming for details on these concepts.","category":"page"},{"location":"overview/#Variants","page":"Overview","title":"Variants","text":"","category":"section"},{"location":"overview/","page":"Overview","title":"Overview","text":"Several variants of each operator are defined:","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"out-of-place in-place (or not) out-of-place + primal in-place (or not) + primal\nderivative derivative!! value_and_derivative value_and_derivative!!\ngradient gradient!! value_and_gradient value_and_gradient!!\njacobian jacobian!! value_and_jacobian value_and_jacobian!!\npushforward pushforward!! value_and_pushforward value_and_pushforward!!\npullback pullback!! value_and_pullback value_and_pullback!!","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"warning: Warning\nThe \"bang-bang\" syntactic convention !! signals that some of the arguments can be mutated, but they do not have to be. Users should not rely on mutation, but instead recover the function output and work from there.grad = gradient!!(f, grad, backend, x) # good\ngradient!!(f, grad, backend, x) # bad, cause `grad` may not have changed","category":"page"},{"location":"overview/#Second-order","page":"Overview","title":"Second order","text":"","category":"section"},{"location":"overview/","page":"Overview","title":"Overview","text":"Second-order differentiation is also supported, with the following operators:","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"operator input x output y result type result shape\nsecond_derivative Number Any same as y size(y)\nhvp Any Number same as x size(x)\nhessian AbstractArray Number AbstractMatrix (length(x), length(x))","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"danger: Danger\nThis is an experimental functionality, use at your own risk.","category":"page"},{"location":"overview/#Preparation","page":"Overview","title":"Preparation","text":"","category":"section"},{"location":"overview/","page":"Overview","title":"Overview","text":"In many cases, automatic differentiation can be accelerated if the function has been run at least once (e.g. to record a tape) and if some cache objects are provided. This is a backend-specific procedure, but we expose a common syntax to achieve it.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"operator preparation function\nderivative prepare_derivative\ngradient prepare_gradient\njacobian prepare_jacobian\nsecond_derivative prepare_second_derivative\nhessian prepare_hessian\npushforward prepare_pushforward\npullback prepare_pullback\nhvp prepare_hvp","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"If you run prepare_operator(backend, f, x), it will create an object called extras containing the necessary information to speed up operator and its variants. This information is specific to backend and f, as well as the type and size of the input x, but it should work with different values of x.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"You can then call operator(backend, f, similar_x, extras), which should be faster than operator(backend, f, similar_x). This is especially worth it if you plan to call operator several times in similar settings: you can think of it as a warm up.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"By default, all the preparation functions return nothing. We do not make any guarantees on their implementation for each backend, or on the performance gains that can be expected.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"warning: Warning\nWe haven't fully figured out what must happen when an extras object is prepared for a specific operator but then given to a lower-level one (i.e. prepare it for jacobian but then give it to pushforward inside jacobian).","category":"page"},{"location":"overview/#Multiple-inputs/outputs","page":"Overview","title":"Multiple inputs/outputs","text":"","category":"section"},{"location":"overview/","page":"Overview","title":"Overview","text":"Restricting the API to one input and one output has many coding advantages, but it is not very flexible. If you need more than that, use ComponentArrays.jl to wrap several objects inside a single ComponentVector.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"CollapsedDocStrings = true","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"using ADTypes, DifferentiationInterface\nusing DifferentiationInterface.DifferentiationTest: backend_string\nimport Markdown\nimport Enzyme, FastDifferentiation, FiniteDiff, FiniteDifferences, ForwardDiff, PolyesterForwardDiff, ReverseDiff, Tracker, Zygote\n\nfunction all_backends()\n return [\n AutoDiffractor(),\n AutoEnzyme(Enzyme.Forward),\n AutoEnzyme(Enzyme.Reverse),\n AutoFastDifferentiation(),\n AutoFiniteDiff(),\n AutoFiniteDifferences(FiniteDifferences.central_fdm(5, 1)),\n AutoForwardDiff(),\n AutoPolyesterForwardDiff(; chunksize=2),\n AutoReverseDiff(),\n AutoTracker(),\n AutoZygote(),\n ]\nend\n\nfunction all_backends_without_enzyme()\n return filter(all_backends()) do b\n !isa(b, AutoEnzyme)\n end\nend","category":"page"},{"location":"backends/#Backends","page":"Backends","title":"Backends","text":"","category":"section"},{"location":"backends/#Types","page":"Backends","title":"Types","text":"","category":"section"},{"location":"backends/","page":"Backends","title":"Backends","text":"Most backend choices are defined by ADTypes.jl.","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"warning: Warning\nOnly the backends listed here are supported by DifferentiationInterface.jl, even though ADTypes.jl defines more.","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"AutoChainRules\nAutoDiffractor\nAutoEnzyme\nAutoForwardDiff\nAutoForwardDiff()\nAutoFiniteDiff\nAutoFiniteDifferences\nAutoPolyesterForwardDiff\nAutoPolyesterForwardDiff()\nAutoReverseDiff\nAutoTracker\nAutoZygote","category":"page"},{"location":"backends/#ADTypes.AutoChainRules","page":"Backends","title":"ADTypes.AutoChainRules","text":"AutoChainRules{RC}\n\nChooses any AD library based on ChainRulesCore.jl, given an appropriate RuleConfig object.\n\nFields\n\nruleconfig::RC\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoDiffractor","page":"Backends","title":"ADTypes.AutoDiffractor","text":"AutoDiffractor\n\nChooses Diffractor.jl.\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoEnzyme","page":"Backends","title":"ADTypes.AutoEnzyme","text":"AutoEnzyme(Enzyme.Forward)\nAutoEnzyme(Enzyme.Reverse)\n\nConstruct a forward or reverse mode AutoEnzyme backend.\n\n\n\n\n\nAutoEnzyme{M}\n\nChooses Enzyme.jl.\n\nFields\n\nmode::M = nothing\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoForwardDiff","page":"Backends","title":"ADTypes.AutoForwardDiff","text":"AutoForwardDiff{chunksize,T}\n\nChooses ForwardDiff.jl.\n\nFields\n\ntag::T\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoForwardDiff-Tuple{}","page":"Backends","title":"ADTypes.AutoForwardDiff","text":"AutoForwardDiff(; chunksize = nothing, tag = nothing)\n\nConstructor.\n\n\n\n\n\n","category":"method"},{"location":"backends/#ADTypes.AutoFiniteDiff","page":"Backends","title":"ADTypes.AutoFiniteDiff","text":"AutoFiniteDiff{T1,T2,T3}\n\nChooses FiniteDiff.jl.\n\nFields\n\nfdtype::T1 = Val(:forward)\nfdjtype::T2 = fdtype\nfdhtype::T3 = Val(:hcentral)\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoFiniteDifferences","page":"Backends","title":"ADTypes.AutoFiniteDifferences","text":"AutoFiniteDifferences{T}\n\nChooses FiniteDifferences.jl.\n\nFields\n\nfdm::T = nothing\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoPolyesterForwardDiff","page":"Backends","title":"ADTypes.AutoPolyesterForwardDiff","text":"AutoPolyesterForwardDiff{chunksize}\n\nChooses PolyesterForwardDiff.jl.\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoPolyesterForwardDiff-Tuple{}","page":"Backends","title":"ADTypes.AutoPolyesterForwardDiff","text":"AutoPolyesterForwardDiff(; chunksize = nothing)\n\nConstructor.\n\n\n\n\n\n","category":"method"},{"location":"backends/#ADTypes.AutoReverseDiff","page":"Backends","title":"ADTypes.AutoReverseDiff","text":"AutoReverseDiff\n\nChooses ReverseDiff.jl.\n\nFields\n\ncompile::Bool = false\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoTracker","page":"Backends","title":"ADTypes.AutoTracker","text":"AutoTracker\n\nChooses Tracker.jl.\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoZygote","page":"Backends","title":"ADTypes.AutoZygote","text":"AutoZygote\n\nChooses Zygote.jl.\n\n\n\n\n\n","category":"type"},{"location":"backends/","page":"Backends","title":"Backends","text":"We also provide a few of our own:","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"AutoFastDifferentiation","category":"page"},{"location":"backends/#DifferentiationInterface.AutoFastDifferentiation","page":"Backends","title":"DifferentiationInterface.AutoFastDifferentiation","text":"AutoFastDifferentiation\n\nChooses FastDifferentiation.jl.\n\n\n\n\n\n","category":"type"},{"location":"backends/#Availability","page":"Backends","title":"Availability","text":"","category":"section"},{"location":"backends/","page":"Backends","title":"Backends","text":"You can use check_available to verify whether a given backend is loaded, like we did below:","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"header = \"| Backend | available |\" # hide\nsubheader = \"|---|---|\" # hide\nrows = map(all_backends()) do backend # hide\n \"| `$(backend_string(backend))` | $(check_available(backend) ? '✓' : '✗') |\" # hide\nend # hide\nMarkdown.parse(join(vcat(header, subheader, rows...), \"\\n\")) # hide","category":"page"},{"location":"backends/#Mutation-support","page":"Backends","title":"Mutation support","text":"","category":"section"},{"location":"backends/","page":"Backends","title":"Backends","text":"All backends are compatible with allocating functions f(x) = y. Only some are compatible with mutating functions f!(y, x) = nothing. You can use check_mutation to check that feature, like we did below:","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"header = \"| Backend | mutation |\" # hide\nsubheader = \"|---|---|\" # hide\nrows = map(all_backends()) do backend # hide\n \"| `$(backend_string(backend))` | $(check_mutation(backend) ? '✓' : '✗') |\" # hide\nend # hide\nMarkdown.parse(join(vcat(header, subheader, rows...), \"\\n\")) # hide","category":"page"},{"location":"backends/#Package-extensions","page":"Backends","title":"Package extensions","text":"","category":"section"},{"location":"backends/","page":"Backends","title":"Backends","text":"CurrentModule = DifferentiationInterface","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"Backend-specific extension content is not part of the public API.","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"Modules = [\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceChainRulesCoreExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceDiffractorExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceEnzymeExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceFastDifferentiationExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceFiniteDiffExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceFiniteDifferencesExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceForwardDiffExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfacePolyesterForwardDiffExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceReverseDiffExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceTrackerExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceZygoteExt)\n]\nFilter = t -> !(t isa Type && t <: ADTypes.AbstractADType)","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"","category":"page"},{"location":"","page":"Home","title":"Home","text":"EditURL = \"https://github.com/gdalle/DifferentiationInterface.jl/blob/main/README.md\"","category":"page"},{"location":"#DifferentiationInterface","page":"Home","title":"DifferentiationInterface","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"(Image: Dev) (Image: Build Status) (Image: Coverage) (Image: Code Style: Blue)","category":"page"},{"location":"","page":"Home","title":"Home","text":"An interface to various automatic differentiation backends in Julia.","category":"page"},{"location":"#Goal","page":"Home","title":"Goal","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"This package provides a backend-agnostic syntax to differentiate functions of the following types:","category":"page"},{"location":"","page":"Home","title":"Home","text":"allocating: f(x) = y\nmutating: f!(y, x) = nothing","category":"page"},{"location":"#Features","page":"Home","title":"Features","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"First and second order operators\nIn-place and out-of-place differentiation\nPreparation mechanism (e.g. to create a config or tape)\nCross-backend testing and benchmarking utilities\nThorough validation on standard inputs and outputs (scalars, vectors, matrices)","category":"page"},{"location":"#Compatibility","page":"Home","title":"Compatibility","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"We support most of the backends defined by ADTypes.jl:","category":"page"},{"location":"","page":"Home","title":"Home","text":"backend object\nChainRulesCore.jl AutoChainRules(ruleconfig)\nDiffractor.jl AutoDiffractor()\nEnzyme.jl AutoEnzyme(Enzyme.Forward) or AutoEnzyme(Enzyme.Reverse)\nFiniteDiff.jl AutoFiniteDiff()\nFiniteDifferences.jl AutoFiniteDifferences(fdm)\nForwardDiff.jl AutoForwardDiff()\nPolyesterForwardDiff.jl AutoPolyesterForwardDiff(; chunksize)\nReverseDiff.jl AutoReverseDiff()\nTracker.jl AutoTracker()\nZygote.jl AutoZygote()","category":"page"},{"location":"","page":"Home","title":"Home","text":"We also provide one additional backend:","category":"page"},{"location":"","page":"Home","title":"Home","text":"backend object\nFastDifferentiation.jl AutoFastDifferentiation()","category":"page"},{"location":"#Example","page":"Home","title":"Example","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"julia> import ADTypes, ForwardDiff\n\njulia> using DifferentiationInterface\n\njulia> backend = ADTypes.AutoForwardDiff();\n\njulia> f(x) = sum(abs2, x);\n\njulia> value_and_gradient(f, backend, [1., 2., 3.])\n(14.0, [2.0, 4.0, 6.0])","category":"page"},{"location":"#Related-packages","page":"Home","title":"Related packages","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"AbstractDifferentiation.jl is the original inspiration for DifferentiationInterface.jl.\nAutoDiffOperators.jl is an attempt to bridge ADTypes.jl with AbstractDifferentiation.jl.","category":"page"},{"location":"","page":"Home","title":"Home","text":"","category":"page"}] +[{"location":"api/","page":"API reference","title":"API reference","text":"CurrentModule = DifferentiationInterface\nCollapsedDocStrings = true","category":"page"},{"location":"api/#API-reference","page":"API reference","title":"API reference","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"DifferentiationInterface","category":"page"},{"location":"api/#DifferentiationInterface.DifferentiationInterface","page":"API reference","title":"DifferentiationInterface.DifferentiationInterface","text":"DifferentiationInterface\n\nAn interface to various automatic differentiation backends in Julia.\n\nExports\n\nAutoFastDifferentiation\nSecondOrder\ncheck_available\ncheck_hessian\ncheck_mutation\nderivative\nderivative!!\ngradient\ngradient!!\nhessian\nhvp\njacobian\njacobian!!\nprepare_derivative\nprepare_gradient\nprepare_hessian\nprepare_hvp\nprepare_jacobian\nprepare_pullback\nprepare_pushforward\nprepare_second_derivative\npullback\npullback!!\npushforward\npushforward!!\nsecond_derivative\nvalue_and_derivative\nvalue_and_derivative!!\nvalue_and_gradient\nvalue_and_gradient!!\nvalue_and_jacobian\nvalue_and_jacobian!!\nvalue_and_pullback\nvalue_and_pullback!!\nvalue_and_pushforward\nvalue_and_pushforward!!\n\n\n\n\n\n","category":"module"},{"location":"api/#Derivative","page":"API reference","title":"Derivative","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"src/derivative.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.derivative!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Number}, Tuple{F, Any, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.derivative!!","text":"derivative!!(f, der, backend, x, [extras]) -> der\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.derivative-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Number}, Tuple{F, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.derivative","text":"derivative(f, backend, x, [extras]) -> der\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_derivative!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Number}, Tuple{F, Any, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_derivative!!","text":"value_and_derivative!!(f, der, backend, x, [extras]) -> (y, der)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_derivative!!-Union{Tuple{F}, Tuple{F, Any, Any, ADTypes.AbstractADType, Number}, Tuple{F, Any, Any, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_derivative!!","text":"value_and_derivative!!(f!, y, der, backend, x, [extras]) -> (y, der)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_derivative-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Number}, Tuple{F, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_derivative","text":"value_and_derivative(f, backend, x, [extras]) -> (y, der)\n\n\n\n\n\n","category":"method"},{"location":"api/#Gradient","page":"API reference","title":"Gradient","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"gradient.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.gradient!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.gradient!!","text":"gradient!!(f, grad, backend, x, [extras]) -> grad\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.gradient-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.gradient","text":"gradient(f, backend, x, [extras]) -> grad\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_gradient!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_gradient!!","text":"value_and_gradient!!(f, grad, backend, x, [extras]) -> (y, grad)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_gradient-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_gradient","text":"value_and_gradient(f, backend, x, [extras]) -> (y, grad)\n\n\n\n\n\n","category":"method"},{"location":"api/#Jacobian","page":"API reference","title":"Jacobian","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"jacobian.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.jacobian!!-Union{Tuple{F}, Tuple{F, AbstractMatrix, ADTypes.AbstractADType, Any}, Tuple{F, AbstractMatrix, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.jacobian!!","text":"jacobian!!(f, jac, backend, x, [extras]) -> jac\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.jacobian-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.jacobian","text":"jacobian(f, backend, x, [extras]) -> jac\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_jacobian!!-Union{Tuple{F}, Tuple{F, AbstractArray, AbstractMatrix, ADTypes.AbstractADType, AbstractArray}, Tuple{F, AbstractArray, AbstractMatrix, ADTypes.AbstractADType, AbstractArray, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_jacobian!!","text":"value_and_jacobian!!(f!, y, jac, backend, x, [extras]) -> (y, jac)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_jacobian!!-Union{Tuple{F}, Tuple{F, AbstractMatrix, ADTypes.AbstractADType, Any}, Tuple{F, AbstractMatrix, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_jacobian!!","text":"value_and_jacobian!!(f, jac, backend, x, [extras]) -> (y, jac)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_jacobian-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_jacobian","text":"value_and_jacobian(f, backend, x, [extras]) -> (y, jac)\n\n\n\n\n\n","category":"method"},{"location":"api/#Second-order","page":"API reference","title":"Second order","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"second_order.jl\", \"second_derivative.jl\", \"hessian.jl\", \"hvp.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.SecondOrder","page":"API reference","title":"DifferentiationInterface.SecondOrder","text":"SecondOrder\n\nCombination of two backends for second-order differentiation.\n\nFields\n\nouter::ADTypes.AbstractADType: backend for the outer differentiation\ninner::ADTypes.AbstractADType: backend for the inner differentiation\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.second_derivative-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Number}, Tuple{F, ADTypes.AbstractADType, Number, Any}} where F","page":"API reference","title":"DifferentiationInterface.second_derivative","text":"second_derivative(f, backend, x, [extras]) -> der2\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.hessian-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.hessian","text":"hessian(f, backend, x, [extras]) -> hess\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.hvp-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.hvp","text":"hvp(f, backend, x, v, [extras]) -> p\n\n\n\n\n\n","category":"method"},{"location":"api/#Primitives","page":"API reference","title":"Primitives","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"pushforward.jl\", \"pullback.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.pushforward!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.pushforward!!","text":"pushforward!!(f, dy, backend, x, dx, [extras]) -> (y, dy)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.pushforward-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.pushforward","text":"pushforward(f, backend, x, dx, [extras]) -> (y, dy)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pushforward!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pushforward!!","text":"value_and_pushforward!!(f, dy, backend, x, dx, [extras]) -> (y, dy)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pushforward!!-Union{Tuple{F}, Tuple{F, Any, Any, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pushforward!!","text":"value_and_pushforward!!(f!, y, dy, backend, x, dx, [extras]) -> (y, dy)\n\ninfo: Info\nRequired primitive for forward mode backends to support mutating functions.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pushforward-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pushforward","text":"value_and_pushforward(f, backend, x, dx, [extras]) -> (y, dy)\n\ninfo: Info\nRequired primitive for forward mode backends to support allocating functions.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.pullback!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.pullback!!","text":"pullback!!(f, dx, backend, x, dy, [extras]) -> dx\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.pullback-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any, Any}, Tuple{F, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.pullback","text":"pullback(f, backend, x, dy, [extras]) -> dx\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pullback!!-Union{Tuple{F}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any}, Tuple{F, Any, ADTypes.AbstractADType, Any, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pullback!!","text":"value_and_pullback!!(f, dx, backend, x, dy, [extras]) -> (y, dx)\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pullback!!-Union{Tuple{F}, Tuple{F, Any, Any, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pullback!!","text":"value_and_pullback!!(f!, y, dx, backend, x, dy, [extras]) -> (y, dx)\n\ninfo: Info\nRequired primitive for reverse mode backends to support mutating functions.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.value_and_pullback-Union{Tuple{F}, Tuple{F, ADTypes.AbstractADType, Any, Any}} where F","page":"API reference","title":"DifferentiationInterface.value_and_pullback","text":"value_and_pullback(f, backend, x, dy, [extras]) -> (y, dx)\n\ninfo: Info\nRequired primitive for reverse mode backends to support allocating functions.\n\n\n\n\n\n","category":"method"},{"location":"api/#Backend-queries","page":"API reference","title":"Backend queries","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"backends.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.check_available-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.check_available","text":"check_available(backend)\n\nCheck whether backend is available by trying a scalar-to-scalar derivative.\n\nwarning: Warning\nMight take a while due to compilation time.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.check_hessian-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.check_hessian","text":"check_hessian(backend)\n\nCheck whether backend supports second order differentiation by trying a hessian.\n\nwarning: Warning\nMight take a while due to compilation time.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.check_mutation-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.check_mutation","text":"check_mutation(backend)\n\nCheck whether backend supports differentiation of mutating functions by trying a jacobian.\n\nwarning: Warning\nMight take a while due to compilation time.\n\n\n\n\n\n","category":"method"},{"location":"api/#Preparation","page":"API reference","title":"Preparation","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPages = [\"prepare.jl\"]","category":"page"},{"location":"api/#DifferentiationInterface.prepare_derivative-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_derivative","text":"prepare_derivative(f, backend, x) -> extras\nprepare_derivative(f!, backend, y, x) -> extras\n\nCreate an extras object that can be given to derivative operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_gradient-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_gradient","text":"prepare_gradient(f, backend, x) -> extras\n\nCreate an extras object that can be given to gradient operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_hessian-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_hessian","text":"prepare_hessian(f, backend, x) -> extras\n\nCreate an extras object that can be given to Hessian operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_hvp-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_hvp","text":"prepare_hvp(f, backend, x) -> extras\n\nCreate an extras object that can be given to Hessian-vector product operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_jacobian-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_jacobian","text":"prepare_jacobian(f, backend, x) -> extras\nprepare_jacobian(f!, backend, y, x) -> extras\n\nCreate an extras object that can be given to Jacobian operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_pullback-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_pullback","text":"prepare_pullback(f, backend, x) -> extras\nprepare_pullback(f!, backend, y, x) -> extras\n\nCreate an extras object that can be given to pullback operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_pushforward-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_pushforward","text":"prepare_pushforward(f, backend, x) -> extras\nprepare_pushforward(f!, backend, y, x) -> extras\n\nCreate an extras object that can be given to pushforward operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.prepare_second_derivative-Tuple{Any, ADTypes.AbstractADType, Any}","page":"API reference","title":"DifferentiationInterface.prepare_second_derivative","text":"prepare_second_derivative(f, backend, x) -> extras\nprepare_second_derivative(f!, backend, y, x) -> extras\n\nCreate an extras object that can be given to second derivative operators.\n\n\n\n\n\n","category":"method"},{"location":"api/#Testing-and-benchmarking","page":"API reference","title":"Testing & benchmarking","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationTest]\nPrivate = false","category":"page"},{"location":"api/#DifferentiationInterface.DifferentiationTest","page":"API reference","title":"DifferentiationInterface.DifferentiationTest","text":"DifferentiationInterface.DifferentiationTest\n\nTesting utilities for DifferentiationInterface.\n\n\n\n\n\n","category":"module"},{"location":"api/#DifferentiationInterface.DifferentiationTest.BenchmarkData","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.BenchmarkData","text":"BenchmarkData\n\nAd-hoc storage type for differentiation benchmarking results. You can turn it into a DataFrame as follows:\n\ndf = DataFrames.DataFrame(pairs(benchmark_data)...)\n\nFields\n\nThese are not part of the public API.\n\nbackend::Vector{String}\nmode::Vector{Type}\noperator::Vector{Function}\nvariant::Vector{Function}\nfunc::Vector{String}\nmutating::Vector{Bool}\ninput_type::Vector{Type}\noutput_type::Vector{Type}\ninput_size::Vector\noutput_size::Vector\nsamples::Vector{Int64}\ntime::Vector{Float64}\nbytes::Vector{Float64}\nallocs::Vector{Float64}\ncompile_fraction::Vector{Float64}\ngc_fraction::Vector{Float64}\nevals::Vector{Float64}\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.DifferentiationTest.Scenario","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.Scenario","text":"Scenario{mutating}\n\nStore a testing scenario composed of a function and its input + output + tangents.\n\nFields\n\nf::Any: function\nx::Any: input\ny::Any: output\ndx::Any: pushforward seed\ndy::Any: pullback seed\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.DifferentiationTest.all_operators-Tuple{}","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.all_operators","text":"all_operators()\n\nList all operators that can be tested with test_differentiation.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.DifferentiationTest.default_scenarios-Tuple{}","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.default_scenarios","text":"default_scenarios()\n\nCreate a vector of Scenarios for testing differentiation. \n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.DifferentiationTest.test_differentiation","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.test_differentiation","text":"test_differentiation(backends, [operators, scenarios]; [kwargs...])\n\nCross-test a list of backends for a list of operators on a list of scenarios, running a variety of different tests.\n\nIf benchmark is false, this returns a TestSet object.\nIf benchmark is true, this returns a BenchmarkData object, which is easy to turn into a DataFrame.\n\nDefault arguments\n\noperators::Vector{Function}: the list [pushforward, pullback,derivative, gradient, jacobian, second_derivative, hvp, hessian]\nscenarios::Vector{Scenario}: the output of default_scenarios()\n\nKeyword arguments\n\nTesting:\n\ncorrectness=true: whether to compare the differentiation results with those given by ForwardDiff.jl\nerror_free=false: whether to run it once and see if it errors\ncall_count=false: whether to check that the function is called the right number of times\ntype_stability=false: whether to check type stability with JET.jl (@test_opt)\nbenchmark=false: whether to run and return a benchmark suite with Chairmarks.jl\nallocations=false: whether to check that the benchmarks are allocation-free\ndetailed=false: whether to print a detailed test set (by scenario) or condensed test set (by operator)\n\nFiltering:\n\ninput_type=Any: restrict scenario inputs to subtypes of this\noutput_type=Any: restrict scenario outputs to subtypes of this\nfirst_order=true: consider first order operators\nsecond_order=true: consider second order operators\nallocating=true: consider operators for allocating functions\nmutating=true: consider operators for mutating functions\nexcluded=Symbol[]: list of excluded operators\n\n\n\n\n\n","category":"function"},{"location":"api/#DifferentiationInterface.DifferentiationTest.test_differentiation-Tuple{ADTypes.AbstractADType, Vararg{Any}}","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.test_differentiation","text":"test_differentiation(\n backend::ADTypes.AbstractADType,\n args...;\n kwargs...\n) -> Union{Test.FallbackTestSet, BenchmarkData, Test.DefaultTestSet}\n\n\nShortcut for a single backend.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.DifferentiationTest.weird_array_scenarios-Tuple{}","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.weird_array_scenarios","text":"weird_array_scenarios()\n\nCreate a vector of Scenarios involving weird types for testing differentiation.\n\n\n\n\n\n","category":"method"},{"location":"api/#Internals","page":"API reference","title":"Internals","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"This is not part of the public API.","category":"page"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationInterface]\nPublic = false\nOrder = [:function, :type]\nFilter = t -> !(t isa Type && t <: ADTypes.AbstractADType)","category":"page"},{"location":"api/#DifferentiationInterface.basisarray-Tuple{ADTypes.AbstractADType, AbstractArray, Any}","page":"API reference","title":"DifferentiationInterface.basisarray","text":"basisarray(backend, a::AbstractArray, i::CartesianIndex)\n\nConstruct the i-th stardard basis array in the vector space of a with element type eltype(a).\n\nNote\n\nIf an AD backend benefits from a more specialized basis array implementation, this function can be extended on the backend type.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.mode-Tuple{ADTypes.AbstractForwardMode}","page":"API reference","title":"DifferentiationInterface.mode","text":"mode(backend)\n\nReturn the AD mode of a backend in a statically predictable way.\n\nThe return value is a Type object chosen among:\n\nADTypes.AbstractForwardMode\nADTypes.AbstractFiniteDifferencesMode\nADTypes.AbstractReverseMode\nADTypes.AbstractSymbolicDifferentiationMode\n\nThis function exists because there are backends (like Enzyme) that can support both forward and reverse mode, which means their ADTypes.jl object does not subtype either.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.supports_mutation-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.supports_mutation","text":"supports_mutation(backend)\n\nReturn MutationSupported or MutationNotSupported in a statically predictable way.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.supports_pullback-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.supports_pullback","text":"supports_pullback(backend)\n\nReturn PullbackSupported or PullbackNotSupported in a statically predictable way.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.supports_pushforward-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.supports_pushforward","text":"supports_pushforward(backend)\n\nReturn PushforwardSupported or PushforwardNotSupported in a statically predictable way.\n\n\n\n\n\n","category":"method"},{"location":"api/#DifferentiationInterface.ForwardOverForward","page":"API reference","title":"DifferentiationInterface.ForwardOverForward","text":"ForwardOverForward\n\nTraits identifying second-order backends that compute HVPs in forward over forward mode (inefficient).\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.ForwardOverReverse","page":"API reference","title":"DifferentiationInterface.ForwardOverReverse","text":"ForwardOverReverse\n\nTraits identifying second-order backends that compute HVPs in forward over reverse mode.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.MutationNotSupported","page":"API reference","title":"DifferentiationInterface.MutationNotSupported","text":"MutationNotSupported\n\nTrait identifying backends that do not support mutating functions f!(y, x).\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.MutationSupported","page":"API reference","title":"DifferentiationInterface.MutationSupported","text":"MutationSupported\n\nTrait identifying backends that support mutating functions f!(y, x).\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.PullbackNotSupported","page":"API reference","title":"DifferentiationInterface.PullbackNotSupported","text":"PullbackNotSupported\n\nTrait identifying backends that do not support efficient pullbacks.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.PullbackSupported","page":"API reference","title":"DifferentiationInterface.PullbackSupported","text":"PullbackSupported\n\nTrait identifying backends that support efficient pullbacks.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.PushforwardNotSupported","page":"API reference","title":"DifferentiationInterface.PushforwardNotSupported","text":"PushforwardNotSupported\n\nTrait identifying backends that do not support efficient pushforwards.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.PushforwardSupported","page":"API reference","title":"DifferentiationInterface.PushforwardSupported","text":"PushforwardSupported\n\nTrait identifying backends that support efficient pushforwards.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.ReverseOverForward","page":"API reference","title":"DifferentiationInterface.ReverseOverForward","text":"ReverseOverForward\n\nTraits identifying second-order backends that compute HVPs in reverse over forward mode.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.ReverseOverReverse","page":"API reference","title":"DifferentiationInterface.ReverseOverReverse","text":"ReverseOverReverse\n\nTraits identifying second-order backends that compute HVPs in reverse over reverse mode.\n\n\n\n\n\n","category":"type"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [DifferentiationTest]\nPublic = false","category":"page"},{"location":"api/#DifferentiationInterface.DifferentiationTest.AutoZeroForward","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.AutoZeroForward","text":"AutoZeroForward <: ADTypes.AbstractForwardMode\n\nTrivial backend that sets all derivatives to zero. Used in testing and benchmarking.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.DifferentiationTest.AutoZeroReverse","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.AutoZeroReverse","text":"AutoZeroReverse <: ADTypes.AbstractReverseMode\n\nTrivial backend that sets all derivatives to zero. Used in testing and benchmarking.\n\n\n\n\n\n","category":"type"},{"location":"api/#DifferentiationInterface.DifferentiationTest.backend_string-Tuple{ADTypes.AbstractADType}","page":"API reference","title":"DifferentiationInterface.DifferentiationTest.backend_string","text":"backend_string(backend)\n\nReturn a shorter string than the full object printing from ADTypes.jl. Might be ambiguous.\n\n\n\n\n\n","category":"method"},{"location":"api/","page":"API reference","title":"API reference","text":"","category":"page"},{"location":"developer/#For-AD-developers","page":"For AD developers","title":"For AD developers","text":"","category":"section"},{"location":"developer/#Backend-requirements","page":"For AD developers","title":"Backend requirements","text":"","category":"section"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"To be usable with DifferentiationInterface.jl, an AD backend needs an object subtyping ADTypes.AbstractADType. In addition, some operators must be defined:","category":"page"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"backend subtype pushforward necessary pullback necessary\nADTypes.AbstractForwardMode yes no\nADTypes.AbstractFiniteDifferencesMode yes no\nADTypes.AbstractReverseMode no yes\nADTypes.AbstractSymbolicDifferentiationMode yes yes","category":"page"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"Every backend we support corresponds to a package extension of DifferentiationInterface.jl (located in the ext subfolder). Advanced users are welcome to code more backends and submit pull requests!","category":"page"},{"location":"developer/#Fallback-call-structure","page":"For AD developers","title":"Fallback call structure","text":"","category":"section"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"For simplicity, we remove value_ in the operator names below.","category":"page"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"note: Edge labels\nFull edges in the following graphs require a single call to the destination. Dotted edges require multiple calls to the destination, the number is indicated above.","category":"page"},{"location":"developer/#Forward-mode,-allocating-functions","page":"For AD developers","title":"Forward mode, allocating functions","text":"","category":"section"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"flowchart LR\n pushforward!! --> pushforward\n derivative --> pushforward\n derivative!! --> pushforward!!\n gradient .-> |n|pushforward\n gradient!! .-> |n|pushforward!!\n jacobian .-> |n|pushforward\n jacobian!! .-> |n|pushforward!!","category":"page"},{"location":"developer/#Reverse-mode,-allocating-functions","page":"For AD developers","title":"Reverse mode, allocating functions","text":"","category":"section"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"flowchart LR\n pullback!! --> pullback\n derivative .-> |m|pullback\n derivative!! .-> |m|pullback!!\n gradient --> pullback\n gradient!! --> pullback!!\n jacobian .-> |m|pullback\n jacobian!! .-> |m|pullback!!","category":"page"},{"location":"developer/","page":"For AD developers","title":"For AD developers","text":"","category":"page"},{"location":"overview/#Overview","page":"Overview","title":"Overview","text":"","category":"section"},{"location":"overview/#operators","page":"Overview","title":"Operators","text":"","category":"section"},{"location":"overview/","page":"Overview","title":"Overview","text":"Depending on the type of input and output, differentiation operators can have various names. Most backends have custom implementations, which we reuse if possible.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"We choose the following terminology for the high-level operators we provide:","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"operator input x output y result type result shape\nderivative Number Any same as y size(y)\ngradient Any Number same as x size(x)\njacobian AbstractArray AbstractArray AbstractMatrix (length(y), length(x))","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"They are all based on the following low-level operators:","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"pushforward (or JVP), to propagate input tangents\npullback (or VJP), to backpropagate output cotangents","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"tip: Tip\nSee the book The Elements of Differentiable Programming for details on these concepts.","category":"page"},{"location":"overview/#Variants","page":"Overview","title":"Variants","text":"","category":"section"},{"location":"overview/","page":"Overview","title":"Overview","text":"Several variants of each operator are defined:","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"out-of-place in-place (or not) out-of-place + primal in-place (or not) + primal\nderivative derivative!! value_and_derivative value_and_derivative!!\ngradient gradient!! value_and_gradient value_and_gradient!!\njacobian jacobian!! value_and_jacobian value_and_jacobian!!\npushforward pushforward!! value_and_pushforward value_and_pushforward!!\npullback pullback!! value_and_pullback value_and_pullback!!","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"warning: Warning\nThe \"bang-bang\" syntactic convention !! signals that some of the arguments can be mutated, but they do not have to be. Users should not rely on mutation, but instead recover the function output and work from there.grad = gradient!!(f, grad, backend, x) # good\ngradient!!(f, grad, backend, x) # bad, cause `grad` may not have changed","category":"page"},{"location":"overview/#Second-order","page":"Overview","title":"Second order","text":"","category":"section"},{"location":"overview/","page":"Overview","title":"Overview","text":"Second-order differentiation is also supported, with the following operators:","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"operator input x output y result type result shape\nsecond_derivative Number Any same as y size(y)\nhvp Any Number same as x size(x)\nhessian AbstractArray Number AbstractMatrix (length(x), length(x))","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"danger: Danger\nThis is an experimental functionality, use at your own risk.","category":"page"},{"location":"overview/#Preparation","page":"Overview","title":"Preparation","text":"","category":"section"},{"location":"overview/","page":"Overview","title":"Overview","text":"In many cases, automatic differentiation can be accelerated if the function has been run at least once (e.g. to record a tape) and if some cache objects are provided. This is a backend-specific procedure, but we expose a common syntax to achieve it.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"operator preparation function\nderivative prepare_derivative\ngradient prepare_gradient\njacobian prepare_jacobian\nsecond_derivative prepare_second_derivative\nhessian prepare_hessian\npushforward prepare_pushforward\npullback prepare_pullback\nhvp prepare_hvp","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"If you run prepare_operator(backend, f, x), it will create an object called extras containing the necessary information to speed up operator and its variants. This information is specific to backend and f, as well as the type and size of the input x, but it should work with different values of x.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"You can then call operator(backend, f, similar_x, extras), which should be faster than operator(backend, f, similar_x). This is especially worth it if you plan to call operator several times in similar settings: you can think of it as a warm up.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"By default, all the preparation functions return nothing. We do not make any guarantees on their implementation for each backend, or on the performance gains that can be expected.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"warning: Warning\nWe haven't fully figured out what must happen when an extras object is prepared for a specific operator but then given to a lower-level one (i.e. prepare it for jacobian but then give it to pushforward inside jacobian).","category":"page"},{"location":"overview/#Multiple-inputs/outputs","page":"Overview","title":"Multiple inputs/outputs","text":"","category":"section"},{"location":"overview/","page":"Overview","title":"Overview","text":"Restricting the API to one input and one output has many coding advantages, but it is not very flexible. If you need more than that, use ComponentArrays.jl to wrap several objects inside a single ComponentVector.","category":"page"},{"location":"overview/","page":"Overview","title":"Overview","text":"","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"CollapsedDocStrings = true","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"using ADTypes, DifferentiationInterface\nusing DifferentiationInterface.DifferentiationTest: backend_string\nimport Markdown\nimport Enzyme, FastDifferentiation, FiniteDiff, FiniteDifferences, ForwardDiff, PolyesterForwardDiff, ReverseDiff, Tracker, Zygote\n\nfunction all_backends()\n return [\n AutoDiffractor(),\n AutoEnzyme(Enzyme.Forward),\n AutoEnzyme(Enzyme.Reverse),\n AutoFastDifferentiation(),\n AutoFiniteDiff(),\n AutoFiniteDifferences(FiniteDifferences.central_fdm(5, 1)),\n AutoForwardDiff(),\n AutoPolyesterForwardDiff(; chunksize=2),\n AutoReverseDiff(),\n AutoTracker(),\n AutoZygote(),\n ]\nend\n\nfunction all_backends_without_enzyme()\n return filter(all_backends()) do b\n !isa(b, AutoEnzyme)\n end\nend","category":"page"},{"location":"backends/#Backends","page":"Backends","title":"Backends","text":"","category":"section"},{"location":"backends/#Types","page":"Backends","title":"Types","text":"","category":"section"},{"location":"backends/","page":"Backends","title":"Backends","text":"Most backend choices are defined by ADTypes.jl.","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"warning: Warning\nOnly the backends listed here are supported by DifferentiationInterface.jl, even though ADTypes.jl defines more.","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"AutoChainRules\nAutoDiffractor\nAutoEnzyme\nAutoForwardDiff\nAutoForwardDiff()\nAutoFiniteDiff\nAutoFiniteDifferences\nAutoPolyesterForwardDiff\nAutoPolyesterForwardDiff()\nAutoReverseDiff\nAutoTracker\nAutoZygote","category":"page"},{"location":"backends/#ADTypes.AutoChainRules","page":"Backends","title":"ADTypes.AutoChainRules","text":"AutoChainRules{RC}\n\nChooses any AD library based on ChainRulesCore.jl, given an appropriate RuleConfig object.\n\nFields\n\nruleconfig::RC\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoDiffractor","page":"Backends","title":"ADTypes.AutoDiffractor","text":"AutoDiffractor\n\nChooses Diffractor.jl.\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoEnzyme","page":"Backends","title":"ADTypes.AutoEnzyme","text":"AutoEnzyme(Enzyme.Forward)\nAutoEnzyme(Enzyme.Reverse)\n\nConstruct a forward or reverse mode AutoEnzyme backend.\n\n\n\n\n\nAutoEnzyme{M}\n\nChooses Enzyme.jl.\n\nFields\n\nmode::M = nothing\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoForwardDiff","page":"Backends","title":"ADTypes.AutoForwardDiff","text":"AutoForwardDiff{chunksize,T}\n\nChooses ForwardDiff.jl.\n\nFields\n\ntag::T\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoForwardDiff-Tuple{}","page":"Backends","title":"ADTypes.AutoForwardDiff","text":"AutoForwardDiff(; chunksize = nothing, tag = nothing)\n\nConstructor.\n\n\n\n\n\n","category":"method"},{"location":"backends/#ADTypes.AutoFiniteDiff","page":"Backends","title":"ADTypes.AutoFiniteDiff","text":"AutoFiniteDiff{T1,T2,T3}\n\nChooses FiniteDiff.jl.\n\nFields\n\nfdtype::T1 = Val(:forward)\nfdjtype::T2 = fdtype\nfdhtype::T3 = Val(:hcentral)\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoFiniteDifferences","page":"Backends","title":"ADTypes.AutoFiniteDifferences","text":"AutoFiniteDifferences{T}\n\nChooses FiniteDifferences.jl.\n\nFields\n\nfdm::T = nothing\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoPolyesterForwardDiff","page":"Backends","title":"ADTypes.AutoPolyesterForwardDiff","text":"AutoPolyesterForwardDiff{chunksize}\n\nChooses PolyesterForwardDiff.jl.\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoPolyesterForwardDiff-Tuple{}","page":"Backends","title":"ADTypes.AutoPolyesterForwardDiff","text":"AutoPolyesterForwardDiff(; chunksize = nothing)\n\nConstructor.\n\n\n\n\n\n","category":"method"},{"location":"backends/#ADTypes.AutoReverseDiff","page":"Backends","title":"ADTypes.AutoReverseDiff","text":"AutoReverseDiff\n\nChooses ReverseDiff.jl.\n\nFields\n\ncompile::Bool = false\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoTracker","page":"Backends","title":"ADTypes.AutoTracker","text":"AutoTracker\n\nChooses Tracker.jl.\n\n\n\n\n\n","category":"type"},{"location":"backends/#ADTypes.AutoZygote","page":"Backends","title":"ADTypes.AutoZygote","text":"AutoZygote\n\nChooses Zygote.jl.\n\n\n\n\n\n","category":"type"},{"location":"backends/","page":"Backends","title":"Backends","text":"We also provide a few of our own:","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"AutoFastDifferentiation","category":"page"},{"location":"backends/#DifferentiationInterface.AutoFastDifferentiation","page":"Backends","title":"DifferentiationInterface.AutoFastDifferentiation","text":"AutoFastDifferentiation\n\nChooses FastDifferentiation.jl.\n\n\n\n\n\n","category":"type"},{"location":"backends/#Availability","page":"Backends","title":"Availability","text":"","category":"section"},{"location":"backends/","page":"Backends","title":"Backends","text":"You can use check_available to verify whether a given backend is loaded, like we did below:","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"header = \"| Backend | available |\" # hide\nsubheader = \"|---|---|\" # hide\nrows = map(all_backends()) do backend # hide\n \"| `$(backend_string(backend))` | $(check_available(backend) ? '✓' : '✗') |\" # hide\nend # hide\nMarkdown.parse(join(vcat(header, subheader, rows...), \"\\n\")) # hide","category":"page"},{"location":"backends/#Mutation-support","page":"Backends","title":"Mutation support","text":"","category":"section"},{"location":"backends/","page":"Backends","title":"Backends","text":"All backends are compatible with allocating functions f(x) = y. Only some are compatible with mutating functions f!(y, x) = nothing. You can use check_mutation to check that feature, like we did below:","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"header = \"| Backend | mutation |\" # hide\nsubheader = \"|---|---|\" # hide\nrows = map(all_backends()) do backend # hide\n \"| `$(backend_string(backend))` | $(check_mutation(backend) ? '✓' : '✗') |\" # hide\nend # hide\nMarkdown.parse(join(vcat(header, subheader, rows...), \"\\n\")) # hide","category":"page"},{"location":"backends/#Package-extensions","page":"Backends","title":"Package extensions","text":"","category":"section"},{"location":"backends/","page":"Backends","title":"Backends","text":"CurrentModule = DifferentiationInterface","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"Backend-specific extension content is not part of the public API.","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"Modules = [\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceChainRulesCoreExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceDiffractorExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceEnzymeExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceFastDifferentiationExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceFiniteDiffExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceFiniteDifferencesExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceForwardDiffExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfacePolyesterForwardDiffExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceReverseDiffExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceTrackerExt),\n Base.get_extension(DifferentiationInterface, :DifferentiationInterfaceZygoteExt)\n]\nFilter = t -> !(t isa Type && t <: ADTypes.AbstractADType)","category":"page"},{"location":"backends/","page":"Backends","title":"Backends","text":"","category":"page"},{"location":"","page":"Home","title":"Home","text":"EditURL = \"https://github.com/gdalle/DifferentiationInterface.jl/blob/main/README.md\"","category":"page"},{"location":"#DifferentiationInterface","page":"Home","title":"DifferentiationInterface","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"(Image: Dev) (Image: Build Status) (Image: Coverage) (Image: Code Style: Blue)","category":"page"},{"location":"","page":"Home","title":"Home","text":"An interface to various automatic differentiation backends in Julia.","category":"page"},{"location":"#Goal","page":"Home","title":"Goal","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"This package provides a backend-agnostic syntax to differentiate functions of the following types:","category":"page"},{"location":"","page":"Home","title":"Home","text":"allocating: f(x) = y\nmutating: f!(y, x) = nothing","category":"page"},{"location":"#Features","page":"Home","title":"Features","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"First and second order operators\nIn-place and out-of-place differentiation\nPreparation mechanism (e.g. to create a config or tape)\nCross-backend testing and benchmarking utilities\nThorough validation on standard inputs and outputs (scalars, vectors, matrices)","category":"page"},{"location":"#Compatibility","page":"Home","title":"Compatibility","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"We support most of the backends defined by ADTypes.jl:","category":"page"},{"location":"","page":"Home","title":"Home","text":"backend object\nChainRulesCore.jl AutoChainRules(ruleconfig)\nDiffractor.jl AutoDiffractor()\nEnzyme.jl AutoEnzyme(Enzyme.Forward) or AutoEnzyme(Enzyme.Reverse)\nFiniteDiff.jl AutoFiniteDiff()\nFiniteDifferences.jl AutoFiniteDifferences(fdm)\nForwardDiff.jl AutoForwardDiff()\nPolyesterForwardDiff.jl AutoPolyesterForwardDiff(; chunksize)\nReverseDiff.jl AutoReverseDiff()\nTracker.jl AutoTracker()\nZygote.jl AutoZygote()","category":"page"},{"location":"","page":"Home","title":"Home","text":"We also provide one additional backend:","category":"page"},{"location":"","page":"Home","title":"Home","text":"backend object\nFastDifferentiation.jl AutoFastDifferentiation()","category":"page"},{"location":"#Example","page":"Home","title":"Example","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"julia> import ADTypes, ForwardDiff\n\njulia> using DifferentiationInterface\n\njulia> backend = ADTypes.AutoForwardDiff();\n\njulia> f(x) = sum(abs2, x);\n\njulia> value_and_gradient(f, backend, [1., 2., 3.])\n(14.0, [2.0, 4.0, 6.0])","category":"page"},{"location":"#Related-packages","page":"Home","title":"Related packages","text":"","category":"section"},{"location":"","page":"Home","title":"Home","text":"AbstractDifferentiation.jl is the original inspiration for DifferentiationInterface.jl.\nAutoDiffOperators.jl is an attempt to bridge ADTypes.jl with AbstractDifferentiation.jl.","category":"page"},{"location":"","page":"Home","title":"Home","text":"","category":"page"}] }