API
DifferentiationInterface — ModuleDifferentiationInterfaceAn interface to various automatic differentiation backends in Julia.
Argument wrappers
DifferentiationInterface.Context — TypeContextAbstract supertype for additional context arguments, which can be passed to differentiation operators after the active input x but are not differentiated.
See also
DifferentiationInterface.Constant — TypeConstantConcrete type of Context argument which is kept constant during differentiation.
Note that an operator can be prepared with an arbitrary value of the constant. However, same-point preparation must occur with the exact value that will be reused later.
Example
julia> using DifferentiationInterface
+API · DifferentiationInterface.jl API
DifferentiationInterface — ModuleDifferentiationInterface
An interface to various automatic differentiation backends in Julia.
sourceArgument wrappers
DifferentiationInterface.Context — TypeContext
Abstract supertype for additional context arguments, which can be passed to differentiation operators after the active input x but are not differentiated.
See also
sourceDifferentiationInterface.Constant — TypeConstant
Concrete type of Context argument which is kept constant during differentiation.
Note that an operator can be prepared with an arbitrary value of the constant. However, same-point preparation must occur with the exact value that will be reused later.
Example
julia> using DifferentiationInterface
julia> import ForwardDiff
@@ -13,31 +13,31 @@
julia> gradient(f, AutoForwardDiff(), [1.0, 2.0], Constant(100))
2-element Vector{Float64}:
200.0
- 400.0
sourceDifferentiationInterface.Cache — TypeCache
Concrete type of Context argument which can be mutated with active values during differentiation.
The initial values present inside the cache do not matter.
sourceFirst order
Pushforward
DifferentiationInterface.prepare_pushforward — Functionprepare_pushforward(f, backend, x, tx, [contexts...]) -> prep
-prepare_pushforward(f!, y, backend, x, tx, [contexts...]) -> prep
Create a prep object that can be given to pushforward and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.prepare_pushforward_same_point — Functionprepare_pushforward_same_point(f, backend, x, tx, [contexts...]) -> prep_same
-prepare_pushforward_same_point(f!, y, backend, x, tx, [contexts...]) -> prep_same
Create an prep_same object that can be given to pushforward and its variants if they are applied at the same point x and with the same contexts.
Warning If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.pushforward — Functionpushforward(f, [prep,] backend, x, tx, [contexts...]) -> ty
-pushforward(f!, y, [prep,] backend, x, tx, [contexts...]) -> ty
Compute the pushforward of the function f at point x with a tuple of tangents tx.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named jvp.
sourceDifferentiationInterface.pushforward! — Functionpushforward!(f, dy, [prep,] backend, x, tx, [contexts...]) -> ty
-pushforward!(f!, y, dy, [prep,] backend, x, tx, [contexts...]) -> ty
Compute the pushforward of the function f at point x with a tuple of tangents tx, overwriting ty.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named jvp!.
sourceDifferentiationInterface.value_and_pushforward — Functionvalue_and_pushforward(f, [prep,] backend, x, tx, [contexts...]) -> (y, ty)
-value_and_pushforward(f!, y, [prep,] backend, x, tx, [contexts...]) -> (y, ty)
Compute the value and the pushforward of the function f at point x with a tuple of tangents tx.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp.
Info Required primitive for forward mode backends.
sourceDifferentiationInterface.value_and_pushforward! — Functionvalue_and_pushforward!(f, dy, [prep,] backend, x, tx, [contexts...]) -> (y, ty)
-value_and_pushforward!(f!, y, dy, [prep,] backend, x, tx, [contexts...]) -> (y, ty)
Compute the value and the pushforward of the function f at point x with a tuple of tangents tx, overwriting ty.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp!.
sourcePullback
DifferentiationInterface.prepare_pullback — Functionprepare_pullback(f, backend, x, ty, [contexts...]) -> prep
-prepare_pullback(f!, y, backend, x, ty, [contexts...]) -> prep
Create a prep object that can be given to pullback and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.prepare_pullback_same_point — Functionprepare_pullback_same_point(f, backend, x, ty, [contexts...]) -> prep_same
-prepare_pullback_same_point(f!, y, backend, x, ty, [contexts...]) -> prep_same
Create an prep_same object that can be given to pullback and its variants if they are applied at the same point x and with the same contexts.
Warning If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.pullback — Functionpullback(f, [prep,] backend, x, ty, [contexts...]) -> tx
-pullback(f!, y, [prep,] backend, x, ty, [contexts...]) -> tx
Compute the pullback of the function f at point x with a tuple of tangents ty.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp.
sourceDifferentiationInterface.pullback! — Functionpullback!(f, dx, [prep,] backend, x, ty, [contexts...]) -> tx
-pullback!(f!, y, dx, [prep,] backend, x, ty, [contexts...]) -> tx
Compute the pullback of the function f at point x with a tuple of tangents ty, overwriting dx.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp!.
sourceDifferentiationInterface.value_and_pullback — Functionvalue_and_pullback(f, [prep,] backend, x, ty, [contexts...]) -> (y, tx)
-value_and_pullback(f!, y, [prep,] backend, x, ty, [contexts...]) -> (y, tx)
Compute the value and the pullback of the function f at point x with a tuple of tangents ty.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp.
Info Required primitive for reverse mode backends.
sourceDifferentiationInterface.value_and_pullback! — Functionvalue_and_pullback!(f, dx, [prep,] backend, x, ty, [contexts...]) -> (y, tx)
-value_and_pullback!(f!, y, dx, [prep,] backend, x, ty, [contexts...]) -> (y, tx)
Compute the value and the pullback of the function f at point x with a tuple of tangents ty, overwriting dx.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp!.
sourceDerivative
DifferentiationInterface.prepare_derivative — Functionprepare_derivative(f, backend, x, [contexts...]) -> prep
-prepare_derivative(f!, y, backend, x, [contexts...]) -> prep
Create a prep object that can be given to derivative and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.derivative — Functionderivative(f, [prep,] backend, x, [contexts...]) -> der
-derivative(f!, y, [prep,] backend, x, [contexts...]) -> der
Compute the derivative of the function f at point x.
To improve performance via operator preparation, refer to prepare_derivative.
sourceDifferentiationInterface.derivative! — Functionderivative!(f, der, [prep,] backend, x, [contexts...]) -> der
-derivative!(f!, y, der, [prep,] backend, x, [contexts...]) -> der
Compute the derivative of the function f at point x, overwriting der.
To improve performance via operator preparation, refer to prepare_derivative.
sourceDifferentiationInterface.value_and_derivative — Functionvalue_and_derivative(f, [prep,] backend, x, [contexts...]) -> (y, der)
-value_and_derivative(f!, y, [prep,] backend, x, [contexts...]) -> (y, der)
Compute the value and the derivative of the function f at point x.
To improve performance via operator preparation, refer to prepare_derivative.
sourceDifferentiationInterface.value_and_derivative! — Functionvalue_and_derivative!(f, der, [prep,] backend, x, [contexts...]) -> (y, der)
-value_and_derivative!(f!, y, der, [prep,] backend, x, [contexts...]) -> (y, der)
Compute the value and the derivative of the function f at point x, overwriting der.
To improve performance via operator preparation, refer to prepare_derivative.
sourceGradient
DifferentiationInterface.prepare_gradient — Functionprepare_gradient(f, backend, x, [contexts...]) -> prep
Create a prep object that can be given to gradient and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
sourceDifferentiationInterface.gradient — Functiongradient(f, [prep,] backend, x, [contexts...]) -> grad
Compute the gradient of the function f at point x.
To improve performance via operator preparation, refer to prepare_gradient.
sourceDifferentiationInterface.gradient! — Functiongradient!(f, grad, [prep,] backend, x, [contexts...]) -> grad
Compute the gradient of the function f at point x, overwriting grad.
To improve performance via operator preparation, refer to prepare_gradient.
sourceDifferentiationInterface.value_and_gradient — Functionvalue_and_gradient(f, [prep,] backend, x, [contexts...]) -> (y, grad)
Compute the value and the gradient of the function f at point x.
To improve performance via operator preparation, refer to prepare_gradient.
sourceDifferentiationInterface.value_and_gradient! — Functionvalue_and_gradient!(f, grad, [prep,] backend, x, [contexts...]) -> (y, grad)
Compute the value and the gradient of the function f at point x, overwriting grad.
To improve performance via operator preparation, refer to prepare_gradient.
sourceJacobian
DifferentiationInterface.prepare_jacobian — Functionprepare_jacobian(f, backend, x, [contexts...]) -> prep
-prepare_jacobian(f!, y, backend, x, [contexts...]) -> prep
Create a prep object that can be given to jacobian and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.jacobian — Functionjacobian(f, [prep,] backend, x, [contexts...]) -> jac
-jacobian(f!, y, [prep,] backend, x, [contexts...]) -> jac
Compute the Jacobian matrix of the function f at point x.
To improve performance via operator preparation, refer to prepare_jacobian.
sourceDifferentiationInterface.jacobian! — Functionjacobian!(f, jac, [prep,] backend, x, [contexts...]) -> jac
-jacobian!(f!, y, jac, [prep,] backend, x, [contexts...]) -> jac
Compute the Jacobian matrix of the function f at point x, overwriting jac.
To improve performance via operator preparation, refer to prepare_jacobian.
sourceDifferentiationInterface.value_and_jacobian — Functionvalue_and_jacobian(f, [prep,] backend, x, [contexts...]) -> (y, jac)
-value_and_jacobian(f!, y, [prep,] backend, x, [contexts...]) -> (y, jac)
Compute the value and the Jacobian matrix of the function f at point x.
To improve performance via operator preparation, refer to prepare_jacobian.
sourceDifferentiationInterface.value_and_jacobian! — Functionvalue_and_jacobian!(f, jac, [prep,] backend, x, [contexts...]) -> (y, jac)
-value_and_jacobian!(f!, y, jac, [prep,] backend, x, [contexts...]) -> (y, jac)
Compute the value and the Jacobian matrix of the function f at point x, overwriting jac.
To improve performance via operator preparation, refer to prepare_jacobian.
sourceSecond order
DifferentiationInterface.SecondOrder — TypeSecondOrder
Combination of two backends for second-order differentiation.
Danger SecondOrder backends do not support first-order operators.
Constructor
SecondOrder(outer_backend, inner_backend)
Fields
outer::AbstractADType: backend for the outer differentiationinner::AbstractADType: backend for the inner differentiation
sourceSecond derivative
DifferentiationInterface.prepare_second_derivative — Functionprepare_second_derivative(f, backend, x, [contexts...]) -> prep
Create a prep object that can be given to second_derivative and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
sourceDifferentiationInterface.second_derivative — Functionsecond_derivative(f, [prep,] backend, x, [contexts...]) -> der2
Compute the second derivative of the function f at point x.
To improve performance via operator preparation, refer to prepare_second_derivative.
sourceDifferentiationInterface.second_derivative! — Functionsecond_derivative!(f, der2, [prep,] backend, x, [contexts...]) -> der2
Compute the second derivative of the function f at point x, overwriting der2.
To improve performance via operator preparation, refer to prepare_second_derivative.
sourceDifferentiationInterface.value_derivative_and_second_derivative — Functionvalue_derivative_and_second_derivative(f, [prep,] backend, x, [contexts...]) -> (y, der, der2)
Compute the value, first derivative and second derivative of the function f at point x.
To improve performance via operator preparation, refer to prepare_second_derivative.
sourceDifferentiationInterface.value_derivative_and_second_derivative! — Functionvalue_derivative_and_second_derivative!(f, der, der2, [prep,] backend, x, [contexts...]) -> (y, der, der2)
Compute the value, first derivative and second derivative of the function f at point x, overwriting der and der2.
To improve performance via operator preparation, refer to prepare_second_derivative.
sourceHessian-vector product
DifferentiationInterface.prepare_hvp — Functionprepare_hvp(f, backend, x, tx, [contexts...]) -> prep
Create a prep object that can be given to hvp and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
sourceDifferentiationInterface.prepare_hvp_same_point — Functionprepare_hvp_same_point(f, backend, x, tx, [contexts...]) -> prep_same
Create an prep_same object that can be given to hvp and its variants if they are applied at the same point x and with the same contexts.
Warning If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again.
sourceDifferentiationInterface.hvp — Functionhvp(f, [prep,] backend, x, tx, [contexts...]) -> tg
Compute the Hessian-vector product of f at point x with a tuple of tangents tx.
To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.
sourceDifferentiationInterface.hvp! — Functionhvp!(f, tg, [prep,] backend, x, tx, [contexts...]) -> tg
Compute the Hessian-vector product of f at point x with a tuple of tangents tx, overwriting tg.
To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.
sourceDifferentiationInterface.gradient_and_hvp — Functiongradient_and_hvp(f, [prep,] backend, x, tx, [contexts...]) -> (grad, tg)
Compute the gradient and the Hessian-vector product of f at point x with a tuple of tangents tx.
To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.
sourceDifferentiationInterface.gradient_and_hvp! — Functiongradient_and_hvp!(f, grad, tg, [prep,] backend, x, tx, [contexts...]) -> (grad, tg)
Compute the gradient and the Hessian-vector product of f at point x with a tuple of tangents tx, overwriting grad and tg.
To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.
sourceHessian
DifferentiationInterface.prepare_hessian — Functionprepare_hessian(f, backend, x, [contexts...]) -> prep
Create a prep object that can be given to hessian and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
sourceDifferentiationInterface.hessian — Functionhessian(f, [prep,] backend, x, [contexts...]) -> hess
Compute the Hessian matrix of the function f at point x.
To improve performance via operator preparation, refer to prepare_hessian.
sourceDifferentiationInterface.hessian! — Functionhessian!(f, hess, [prep,] backend, x, [contexts...]) -> hess
Compute the Hessian matrix of the function f at point x, overwriting hess.
To improve performance via operator preparation, refer to prepare_hessian.
sourceDifferentiationInterface.value_gradient_and_hessian — Functionvalue_gradient_and_hessian(f, [prep,] backend, x, [contexts...]) -> (y, grad, hess)
Compute the value, gradient vector and Hessian matrix of the function f at point x.
To improve performance via operator preparation, refer to prepare_hessian.
sourceDifferentiationInterface.value_gradient_and_hessian! — Functionvalue_gradient_and_hessian!(f, grad, hess, [prep,] backend, x, [contexts...]) -> (y, grad, hess)
Compute the value, gradient vector and Hessian matrix of the function f at point x, overwriting grad and hess.
To improve performance via operator preparation, refer to prepare_hessian.
sourceUtilities
Backend queries
DifferentiationInterface.check_available — Functioncheck_available(backend)
Check whether backend is available (i.e. whether the extension is loaded).
sourceDifferentiationInterface.check_inplace — Functioncheck_inplace(backend)
Check whether backend supports differentiation of in-place functions.
sourceDifferentiationInterface.outer — Functionouter(backend::SecondOrder)
-outer(backend::AbstractADType)
Return the outer backend of a SecondOrder object, tasked with differentiation at the second order.
For any other backend type, this function acts like the identity.
sourceDifferentiationInterface.inner — Functioninner(backend::SecondOrder)
-inner(backend::AbstractADType)
Return the inner backend of a SecondOrder object, tasked with differentiation at the first order.
For any other backend type, this function acts like the identity.
sourceBackend switch
DifferentiationInterface.DifferentiateWith — TypeDifferentiateWith
Function wrapper that enforces differentiation with a "substitute" AD backend, possible different from the "true" AD backend that is called.
For instance, suppose a function f is not differentiable with Zygote because it involves mutation, but you know that it is differentiable with Enzyme. Then f2 = DifferentiateWith(f, AutoEnzyme()) is a new function that behaves like f, except that f2 is differentiable with Zygote (thanks to a chain rule which calls Enzyme under the hood). Moreover, any larger algorithm alg that calls f2 instead of f will also be differentiable with Zygote (as long as f was the only Zygote blocker).
Tip This is mainly relevant for package developers who want to produce differentiable code at low cost, without writing the differentiation rules themselves. If you sprinkle a few DifferentiateWith in places where some AD backends may struggle, end users can pick from a wider variety of packages to differentiate your algorithms.
Warning DifferentiateWith only supports out-of-place functions y = f(x) without additional context arguments. It only makes these functions differentiable if the true backend is either ForwardDiff or compatible with ChainRules. For any other true backend, the differentiation behavior is not altered by DifferentiateWith (it becomes a transparent wrapper).
Fields
f: the function in question, with signature f(x)backend::AbstractADType: the substitute backend to use for differentiation
Note For the substitute AD backend to be called under the hood, its package needs to be loaded in addition to the package of the true AD backend.
Constructor
DifferentiateWith(f, backend)
Example
julia> using DifferentiationInterface
+ 400.0
sourceDifferentiationInterface.Cache — TypeCache
Concrete type of Context argument which can be mutated with active values during differentiation.
The initial values present inside the cache do not matter.
sourceFirst order
Pushforward
DifferentiationInterface.prepare_pushforward — Functionprepare_pushforward(f, backend, x, tx, [contexts...]) -> prep
+prepare_pushforward(f!, y, backend, x, tx, [contexts...]) -> prep
Create a prep object that can be given to pushforward and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.prepare_pushforward_same_point — Functionprepare_pushforward_same_point(f, backend, x, tx, [contexts...]) -> prep_same
+prepare_pushforward_same_point(f!, y, backend, x, tx, [contexts...]) -> prep_same
Create an prep_same object that can be given to pushforward and its variants if they are applied at the same point x and with the same contexts.
Warning If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.pushforward — Functionpushforward(f, [prep,] backend, x, tx, [contexts...]) -> ty
+pushforward(f!, y, [prep,] backend, x, tx, [contexts...]) -> ty
Compute the pushforward of the function f at point x with a tuple of tangents tx.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named jvp.
sourceDifferentiationInterface.pushforward! — Functionpushforward!(f, dy, [prep,] backend, x, tx, [contexts...]) -> ty
+pushforward!(f!, y, dy, [prep,] backend, x, tx, [contexts...]) -> ty
Compute the pushforward of the function f at point x with a tuple of tangents tx, overwriting ty.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named jvp!.
sourceDifferentiationInterface.value_and_pushforward — Functionvalue_and_pushforward(f, [prep,] backend, x, tx, [contexts...]) -> (y, ty)
+value_and_pushforward(f!, y, [prep,] backend, x, tx, [contexts...]) -> (y, ty)
Compute the value and the pushforward of the function f at point x with a tuple of tangents tx.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp.
Info Required primitive for forward mode backends.
sourceDifferentiationInterface.value_and_pushforward! — Functionvalue_and_pushforward!(f, dy, [prep,] backend, x, tx, [contexts...]) -> (y, ty)
+value_and_pushforward!(f!, y, dy, [prep,] backend, x, tx, [contexts...]) -> (y, ty)
Compute the value and the pushforward of the function f at point x with a tuple of tangents tx, overwriting ty.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp!.
sourcePullback
DifferentiationInterface.prepare_pullback — Functionprepare_pullback(f, backend, x, ty, [contexts...]) -> prep
+prepare_pullback(f!, y, backend, x, ty, [contexts...]) -> prep
Create a prep object that can be given to pullback and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.prepare_pullback_same_point — Functionprepare_pullback_same_point(f, backend, x, ty, [contexts...]) -> prep_same
+prepare_pullback_same_point(f!, y, backend, x, ty, [contexts...]) -> prep_same
Create an prep_same object that can be given to pullback and its variants if they are applied at the same point x and with the same contexts.
Warning If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.pullback — Functionpullback(f, [prep,] backend, x, ty, [contexts...]) -> tx
+pullback(f!, y, [prep,] backend, x, ty, [contexts...]) -> tx
Compute the pullback of the function f at point x with a tuple of tangents ty.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp.
sourceDifferentiationInterface.pullback! — Functionpullback!(f, dx, [prep,] backend, x, ty, [contexts...]) -> tx
+pullback!(f!, y, dx, [prep,] backend, x, ty, [contexts...]) -> tx
Compute the pullback of the function f at point x with a tuple of tangents ty, overwriting dx.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp!.
sourceDifferentiationInterface.value_and_pullback — Functionvalue_and_pullback(f, [prep,] backend, x, ty, [contexts...]) -> (y, tx)
+value_and_pullback(f!, y, [prep,] backend, x, ty, [contexts...]) -> (y, tx)
Compute the value and the pullback of the function f at point x with a tuple of tangents ty.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp.
Info Required primitive for reverse mode backends.
sourceDifferentiationInterface.value_and_pullback! — Functionvalue_and_pullback!(f, dx, [prep,] backend, x, ty, [contexts...]) -> (y, tx)
+value_and_pullback!(f!, y, dx, [prep,] backend, x, ty, [contexts...]) -> (y, tx)
Compute the value and the pullback of the function f at point x with a tuple of tangents ty, overwriting dx.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp!.
sourceDerivative
DifferentiationInterface.prepare_derivative — Functionprepare_derivative(f, backend, x, [contexts...]) -> prep
+prepare_derivative(f!, y, backend, x, [contexts...]) -> prep
Create a prep object that can be given to derivative and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.derivative — Functionderivative(f, [prep,] backend, x, [contexts...]) -> der
+derivative(f!, y, [prep,] backend, x, [contexts...]) -> der
Compute the derivative of the function f at point x.
To improve performance via operator preparation, refer to prepare_derivative.
sourceDifferentiationInterface.derivative! — Functionderivative!(f, der, [prep,] backend, x, [contexts...]) -> der
+derivative!(f!, y, der, [prep,] backend, x, [contexts...]) -> der
Compute the derivative of the function f at point x, overwriting der.
To improve performance via operator preparation, refer to prepare_derivative.
sourceDifferentiationInterface.value_and_derivative — Functionvalue_and_derivative(f, [prep,] backend, x, [contexts...]) -> (y, der)
+value_and_derivative(f!, y, [prep,] backend, x, [contexts...]) -> (y, der)
Compute the value and the derivative of the function f at point x.
To improve performance via operator preparation, refer to prepare_derivative.
sourceDifferentiationInterface.value_and_derivative! — Functionvalue_and_derivative!(f, der, [prep,] backend, x, [contexts...]) -> (y, der)
+value_and_derivative!(f!, y, der, [prep,] backend, x, [contexts...]) -> (y, der)
Compute the value and the derivative of the function f at point x, overwriting der.
To improve performance via operator preparation, refer to prepare_derivative.
sourceGradient
DifferentiationInterface.prepare_gradient — Functionprepare_gradient(f, backend, x, [contexts...]) -> prep
Create a prep object that can be given to gradient and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
sourceDifferentiationInterface.gradient — Functiongradient(f, [prep,] backend, x, [contexts...]) -> grad
Compute the gradient of the function f at point x.
To improve performance via operator preparation, refer to prepare_gradient.
sourceDifferentiationInterface.gradient! — Functiongradient!(f, grad, [prep,] backend, x, [contexts...]) -> grad
Compute the gradient of the function f at point x, overwriting grad.
To improve performance via operator preparation, refer to prepare_gradient.
sourceDifferentiationInterface.value_and_gradient — Functionvalue_and_gradient(f, [prep,] backend, x, [contexts...]) -> (y, grad)
Compute the value and the gradient of the function f at point x.
To improve performance via operator preparation, refer to prepare_gradient.
sourceDifferentiationInterface.value_and_gradient! — Functionvalue_and_gradient!(f, grad, [prep,] backend, x, [contexts...]) -> (y, grad)
Compute the value and the gradient of the function f at point x, overwriting grad.
To improve performance via operator preparation, refer to prepare_gradient.
sourceJacobian
DifferentiationInterface.prepare_jacobian — Functionprepare_jacobian(f, backend, x, [contexts...]) -> prep
+prepare_jacobian(f!, y, backend, x, [contexts...]) -> prep
Create a prep object that can be given to jacobian and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. For in-place functions, y is mutated by f! during preparation.
sourceDifferentiationInterface.jacobian — Functionjacobian(f, [prep,] backend, x, [contexts...]) -> jac
+jacobian(f!, y, [prep,] backend, x, [contexts...]) -> jac
Compute the Jacobian matrix of the function f at point x.
To improve performance via operator preparation, refer to prepare_jacobian.
sourceDifferentiationInterface.jacobian! — Functionjacobian!(f, jac, [prep,] backend, x, [contexts...]) -> jac
+jacobian!(f!, y, jac, [prep,] backend, x, [contexts...]) -> jac
Compute the Jacobian matrix of the function f at point x, overwriting jac.
To improve performance via operator preparation, refer to prepare_jacobian.
sourceDifferentiationInterface.value_and_jacobian — Functionvalue_and_jacobian(f, [prep,] backend, x, [contexts...]) -> (y, jac)
+value_and_jacobian(f!, y, [prep,] backend, x, [contexts...]) -> (y, jac)
Compute the value and the Jacobian matrix of the function f at point x.
To improve performance via operator preparation, refer to prepare_jacobian.
sourceDifferentiationInterface.value_and_jacobian! — Functionvalue_and_jacobian!(f, jac, [prep,] backend, x, [contexts...]) -> (y, jac)
+value_and_jacobian!(f!, y, jac, [prep,] backend, x, [contexts...]) -> (y, jac)
Compute the value and the Jacobian matrix of the function f at point x, overwriting jac.
To improve performance via operator preparation, refer to prepare_jacobian.
sourceSecond order
DifferentiationInterface.SecondOrder — TypeSecondOrder
Combination of two backends for second-order differentiation.
Danger SecondOrder backends do not support first-order operators.
Constructor
SecondOrder(outer_backend, inner_backend)
Fields
outer::AbstractADType: backend for the outer differentiationinner::AbstractADType: backend for the inner differentiation
sourceSecond derivative
DifferentiationInterface.prepare_second_derivative — Functionprepare_second_derivative(f, backend, x, [contexts...]) -> prep
Create a prep object that can be given to second_derivative and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
sourceDifferentiationInterface.second_derivative — Functionsecond_derivative(f, [prep,] backend, x, [contexts...]) -> der2
Compute the second derivative of the function f at point x.
To improve performance via operator preparation, refer to prepare_second_derivative.
sourceDifferentiationInterface.second_derivative! — Functionsecond_derivative!(f, der2, [prep,] backend, x, [contexts...]) -> der2
Compute the second derivative of the function f at point x, overwriting der2.
To improve performance via operator preparation, refer to prepare_second_derivative.
sourceDifferentiationInterface.value_derivative_and_second_derivative — Functionvalue_derivative_and_second_derivative(f, [prep,] backend, x, [contexts...]) -> (y, der, der2)
Compute the value, first derivative and second derivative of the function f at point x.
To improve performance via operator preparation, refer to prepare_second_derivative.
sourceDifferentiationInterface.value_derivative_and_second_derivative! — Functionvalue_derivative_and_second_derivative!(f, der, der2, [prep,] backend, x, [contexts...]) -> (y, der, der2)
Compute the value, first derivative and second derivative of the function f at point x, overwriting der and der2.
To improve performance via operator preparation, refer to prepare_second_derivative.
sourceHessian-vector product
DifferentiationInterface.prepare_hvp — Functionprepare_hvp(f, backend, x, tx, [contexts...]) -> prep
Create a prep object that can be given to hvp and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
sourceDifferentiationInterface.prepare_hvp_same_point — Functionprepare_hvp_same_point(f, backend, x, tx, [contexts...]) -> prep_same
Create an prep_same object that can be given to hvp and its variants if they are applied at the same point x and with the same contexts.
Warning If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again.
sourceDifferentiationInterface.hvp — Functionhvp(f, [prep,] backend, x, tx, [contexts...]) -> tg
Compute the Hessian-vector product of f at point x with a tuple of tangents tx.
To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.
sourceDifferentiationInterface.hvp! — Functionhvp!(f, tg, [prep,] backend, x, tx, [contexts...]) -> tg
Compute the Hessian-vector product of f at point x with a tuple of tangents tx, overwriting tg.
To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.
sourceDifferentiationInterface.gradient_and_hvp — Functiongradient_and_hvp(f, [prep,] backend, x, tx, [contexts...]) -> (grad, tg)
Compute the gradient and the Hessian-vector product of f at point x with a tuple of tangents tx.
To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.
sourceDifferentiationInterface.gradient_and_hvp! — Functiongradient_and_hvp!(f, grad, tg, [prep,] backend, x, tx, [contexts...]) -> (grad, tg)
Compute the gradient and the Hessian-vector product of f at point x with a tuple of tangents tx, overwriting grad and tg.
To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.
sourceHessian
DifferentiationInterface.prepare_hessian — Functionprepare_hessian(f, backend, x, [contexts...]) -> prep
Create a prep object that can be given to hessian and its variants.
Warning If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
sourceDifferentiationInterface.hessian — Functionhessian(f, [prep,] backend, x, [contexts...]) -> hess
Compute the Hessian matrix of the function f at point x.
To improve performance via operator preparation, refer to prepare_hessian.
sourceDifferentiationInterface.hessian! — Functionhessian!(f, hess, [prep,] backend, x, [contexts...]) -> hess
Compute the Hessian matrix of the function f at point x, overwriting hess.
To improve performance via operator preparation, refer to prepare_hessian.
sourceDifferentiationInterface.value_gradient_and_hessian — Functionvalue_gradient_and_hessian(f, [prep,] backend, x, [contexts...]) -> (y, grad, hess)
Compute the value, gradient vector and Hessian matrix of the function f at point x.
To improve performance via operator preparation, refer to prepare_hessian.
sourceDifferentiationInterface.value_gradient_and_hessian! — Functionvalue_gradient_and_hessian!(f, grad, hess, [prep,] backend, x, [contexts...]) -> (y, grad, hess)
Compute the value, gradient vector and Hessian matrix of the function f at point x, overwriting grad and hess.
To improve performance via operator preparation, refer to prepare_hessian.
sourceUtilities
Backend queries
DifferentiationInterface.check_available — Functioncheck_available(backend)
Check whether backend is available (i.e. whether the extension is loaded).
sourceDifferentiationInterface.check_inplace — Functioncheck_inplace(backend)
Check whether backend supports differentiation of in-place functions.
sourceDifferentiationInterface.outer — Functionouter(backend::SecondOrder)
+outer(backend::AbstractADType)
Return the outer backend of a SecondOrder object, tasked with differentiation at the second order.
For any other backend type, this function acts like the identity.
sourceDifferentiationInterface.inner — Functioninner(backend::SecondOrder)
+inner(backend::AbstractADType)
Return the inner backend of a SecondOrder object, tasked with differentiation at the first order.
For any other backend type, this function acts like the identity.
sourceBackend switch
DifferentiationInterface.DifferentiateWith — TypeDifferentiateWith
Function wrapper that enforces differentiation with a "substitute" AD backend, possible different from the "true" AD backend that is called.
For instance, suppose a function f is not differentiable with Zygote because it involves mutation, but you know that it is differentiable with Enzyme. Then f2 = DifferentiateWith(f, AutoEnzyme()) is a new function that behaves like f, except that f2 is differentiable with Zygote (thanks to a chain rule which calls Enzyme under the hood). Moreover, any larger algorithm alg that calls f2 instead of f will also be differentiable with Zygote (as long as f was the only Zygote blocker).
Tip This is mainly relevant for package developers who want to produce differentiable code at low cost, without writing the differentiation rules themselves. If you sprinkle a few DifferentiateWith in places where some AD backends may struggle, end users can pick from a wider variety of packages to differentiate your algorithms.
Warning DifferentiateWith only supports out-of-place functions y = f(x) without additional context arguments. It only makes these functions differentiable if the true backend is either ForwardDiff or compatible with ChainRules. For any other true backend, the differentiation behavior is not altered by DifferentiateWith (it becomes a transparent wrapper).
Fields
f: the function in question, with signature f(x)backend::AbstractADType: the substitute backend to use for differentiation
Note For the substitute AD backend to be called under the hood, its package needs to be loaded in addition to the package of the true AD backend.
Constructor
DifferentiateWith(f, backend)
Example
julia> using DifferentiationInterface
julia> import FiniteDiff, ForwardDiff, Zygote
@@ -62,7 +62,7 @@
julia> Zygote.gradient(alg, [3.0, 5.0])[1]
2-element Vector{Float64}:
42.0
- 70.0
sourceSparsity detection
DifferentiationInterface.DenseSparsityDetector — TypeDenseSparsityDetector
Sparsity pattern detector satisfying the detection API of ADTypes.jl.
The nonzeros in a Jacobian or Hessian are detected by computing the relevant matrix with dense AD, and thresholding the entries with a given tolerance (which can be numerically inaccurate). This process can be very slow, and should only be used if its output can be exploited multiple times to compute many sparse matrices.
Danger In general, the sparsity pattern you obtain can depend on the provided input x. If you want to reuse the pattern, make sure that it is input-agnostic.
Warning DenseSparsityDetector functionality is now located in a package extension, please load the SparseArrays.jl standard library before you use it.
Fields
backend::AbstractADType is the dense AD backend used under the hoodatol::Float64 is the minimum magnitude of a matrix entry to be considered nonzero
Constructor
DenseSparsityDetector(backend; atol, method=:iterative)
The keyword argument method::Symbol can be either:
:iterative: compute the matrix in a sequence of matrix-vector products (memory-efficient):direct: compute the matrix all at once (memory-hungry but sometimes faster).
Note that the constructor is type-unstable because method ends up being a type parameter of the DenseSparsityDetector object (this is not part of the API and might change).
Examples
using ADTypes, DifferentiationInterface, SparseArrays
+ 70.0
sourceSparsity detection
DifferentiationInterface.DenseSparsityDetector — TypeDenseSparsityDetector
Sparsity pattern detector satisfying the detection API of ADTypes.jl.
The nonzeros in a Jacobian or Hessian are detected by computing the relevant matrix with dense AD, and thresholding the entries with a given tolerance (which can be numerically inaccurate). This process can be very slow, and should only be used if its output can be exploited multiple times to compute many sparse matrices.
Danger In general, the sparsity pattern you obtain can depend on the provided input x. If you want to reuse the pattern, make sure that it is input-agnostic.
Warning DenseSparsityDetector functionality is now located in a package extension, please load the SparseArrays.jl standard library before you use it.
Fields
backend::AbstractADType is the dense AD backend used under the hoodatol::Float64 is the minimum magnitude of a matrix entry to be considered nonzero
Constructor
DenseSparsityDetector(backend; atol, method=:iterative)
The keyword argument method::Symbol can be either:
:iterative: compute the matrix in a sequence of matrix-vector products (memory-efficient):direct: compute the matrix all at once (memory-hungry but sometimes faster).
Note that the constructor is type-unstable because method ends up being a type parameter of the DenseSparsityDetector object (this is not part of the API and might change).
Examples
using ADTypes, DifferentiationInterface, SparseArrays
import ForwardDiff
detector = DenseSparsityDetector(AutoForwardDiff(); atol=1e-5, method=:direct)
@@ -85,13 +85,13 @@
# output
1×2 SparseMatrixCSC{Bool, Int64} with 1 stored entry:
- 1 ⋅
sourceInternals
The following is not part of the public API.
DifferentiationInterface.AutoZeroForward — TypeAutoZeroForward <: ADTypes.AbstractADType
Trivial backend that sets all derivatives to zero. Used in testing and benchmarking.
sourceDifferentiationInterface.AutoZeroReverse — TypeAutoZeroReverse <: ADTypes.AbstractADType
Trivial backend that sets all derivatives to zero. Used in testing and benchmarking.
sourceDifferentiationInterface.BatchSizeSettings — TypeBatchSizeSettings{B,singlebatch,aligned}
Configuration for the batch size deduced from a backend and a sample array of length N.
Type parameters
B::Int: batch sizesinglebatch::Bool: whether B > Naligned::Bool: whether N % B == 0
Fields
N::Int: array lengthA::Int: number of batches A = div(N, B, RoundUp)B_last::Int: size of the last batch (if aligned is false)
sourceDifferentiationInterface.DerivativePrep — TypeDerivativePrep
Abstract type for additional information needed by derivative and its variants.
sourceDifferentiationInterface.ForwardOverForward — TypeForwardOverForward
Traits identifying second-order backends that compute HVPs in forward over forward mode (inefficient).
sourceDifferentiationInterface.ForwardOverReverse — TypeForwardOverReverse
Traits identifying second-order backends that compute HVPs in forward over reverse mode.
sourceDifferentiationInterface.GradientPrep — TypeGradientPrep
Abstract type for additional information needed by gradient and its variants.
sourceDifferentiationInterface.HVPPrep — TypeHVPPrep
Abstract type for additional information needed by hvp and its variants.
sourceDifferentiationInterface.HessianPrep — TypeHessianPrep
Abstract type for additional information needed by hessian and its variants.
sourceDifferentiationInterface.InPlaceNotSupported — TypeInPlaceNotSupported
Trait identifying backends that do not support in-place functions f!(y, x).
sourceDifferentiationInterface.InPlaceSupported — TypeInPlaceSupported
Trait identifying backends that support in-place functions f!(y, x).
sourceDifferentiationInterface.JacobianPrep — TypeJacobianPrep
Abstract type for additional information needed by jacobian and its variants.
sourceDifferentiationInterface.PullbackFast — TypePullbackFast
Trait identifying backends that support efficient pullbacks.
sourceDifferentiationInterface.PullbackPrep — TypePullbackPrep
Abstract type for additional information needed by pullback and its variants.
sourceDifferentiationInterface.PullbackSlow — TypePullbackSlow
Trait identifying backends that do not support efficient pullbacks.
sourceDifferentiationInterface.PushforwardFast — TypePushforwardFast
Trait identifying backends that support efficient pushforwards.
sourceDifferentiationInterface.PushforwardPrep — TypePushforwardPrep
Abstract type for additional information needed by pushforward and its variants.
sourceDifferentiationInterface.PushforwardSlow — TypePushforwardSlow
Trait identifying backends that do not support efficient pushforwards.
sourceDifferentiationInterface.ReverseOverForward — TypeReverseOverForward
Traits identifying second-order backends that compute HVPs in reverse over forward mode.
sourceDifferentiationInterface.ReverseOverReverse — TypeReverseOverReverse
Traits identifying second-order backends that compute HVPs in reverse over reverse mode.
sourceDifferentiationInterface.SecondDerivativePrep — TypeSecondDerivativePrep
Abstract type for additional information needed by second_derivative and its variants.
sourceADTypes.mode — Methodmode(backend::SecondOrder)
Return the outer mode of the second-order backend.
sourceDifferentiationInterface.basis — Methodbasis(backend, a::AbstractArray, i)
Construct the i-th standard basis array in the vector space of a with element type eltype(a).
Note
If an AD backend benefits from a more specialized basis array implementation, this function can be extended on the backend type.
sourceDifferentiationInterface.inplace_support — Methodinplace_support(backend)
Return InPlaceSupported or InPlaceNotSupported in a statically predictable way.
sourceDifferentiationInterface.multibasis — Methodmultibasis(backend, a::AbstractArray, inds::AbstractVector)
Construct the sum of the i-th standard basis arrays in the vector space of a with element type eltype(a), for all i ∈ inds.
Note
If an AD backend benefits from a more specialized basis array implementation, this function can be extended on the backend type.
sourceDifferentiationInterface.prepare!_derivative — Functionprepare!_derivative(f, prep, backend, x, [contexts...]) -> new_prep
-prepare!_derivative(f!, y, prep, backend, x, [contexts...]) -> new_prep
Same behavior as prepare_derivative but can modify an existing prep object to avoid some allocations.
There is no guarantee that prep will be mutated, or that performance will be improved compared to preparation from scratch.
Danger For efficiency, this function needs to rely on backend package internals, therefore it not protected by semantic versioning.
sourceDifferentiationInterface.prepare!_gradient — Functionprepare!_gradient(f, prep, backend, x, [contexts...]) -> new_prep
Same behavior as prepare_gradient but can modify an existing prep object to avoid some allocations.
There is no guarantee that prep will be mutated, or that performance will be improved compared to preparation from scratch.
Danger For efficiency, this function needs to rely on backend package internals, therefore it not protected by semantic versioning.
sourceDifferentiationInterface.prepare!_hessian — Functionprepare!_hessian(f, backend, x, [contexts...]) -> new_prep
Same behavior as prepare_hessian but can modify an existing prep object to avoid some allocations.
There is no guarantee that prep will be mutated, or that performance will be improved compared to preparation from scratch.
Danger For efficiency, this function needs to rely on backend package internals, therefore it not protected by semantic versioning.
sourceDifferentiationInterface.prepare!_hvp — Functionprepare!_hvp(f, backend, x, tx, [contexts...]) -> new_prep
Same behavior as prepare_hvp but can modify an existing prep object to avoid some allocations.
There is no guarantee that prep will be mutated, or that performance will be improved compared to preparation from scratch.
Danger For efficiency, this function needs to rely on backend package internals, therefore it not protected by semantic versioning.
sourceDifferentiationInterface.prepare!_jacobian — Functionprepare!_jacobian(f, prep, backend, x, [contexts...]) -> new_prep
-prepare!_jacobian(f!, y, prep, backend, x, [contexts...]) -> new_prep
Same behavior as prepare_jacobian but can modify an existing prep object to avoid some allocations.
There is no guarantee that prep will be mutated, or that performance will be improved compared to preparation from scratch.
Danger For efficiency, this function needs to rely on backend package internals, therefore it not protected by semantic versioning.
sourceDifferentiationInterface.prepare!_pullback — Functionprepare!_pullback(f, prep, backend, x, ty, [contexts...]) -> new_prep
-prepare!_pullback(f!, y, prep, backend, x, ty, [contexts...]) -> new_prep
Same behavior as prepare_pullback but can modify an existing prep object to avoid some allocations.
There is no guarantee that prep will be mutated, or that performance will be improved compared to preparation from scratch.
Danger For efficiency, this function needs to rely on backend package internals, therefore it not protected by semantic versioning.
sourceDifferentiationInterface.prepare!_pushforward — Functionprepare!_pushforward(f, prep, backend, x, tx, [contexts...]) -> new_prep
-prepare!_pushforward(f!, y, prep, backend, x, tx, [contexts...]) -> new_prep
Same behavior as prepare_pushforward but can modify an existing prep object to avoid some allocations.
There is no guarantee that prep will be mutated, or that performance will be improved compared to preparation from scratch.
Danger For efficiency, this function needs to rely on backend package internals, therefore it not protected by semantic versioning.
sourceDifferentiationInterface.pullback_performance — Methodpullback_performance(backend)
Return PullbackFast or PullbackSlow in a statically predictable way.
sourceDifferentiationInterface.pushforward_performance — Methodpushforward_performance(backend)
Return PushforwardFast or PushforwardSlow in a statically predictable way.
sourceSettings
This document was generated with Documenter.jl version 1.7.0 on Monday 28 October 2024. Using Julia version 1.11.1.
Settings
This document was generated with Documenter.jl version 1.7.0 on Monday 28 October 2024. Using Julia version 1.11.1.