API

DifferentiationInterface — Module

DifferentiationInterface

An interface to various automatic differentiation backends in Julia.

source

DifferentiationInterface.Context — Type

Context

Abstract supertype for additional context arguments, which can be passed to differentiation operators after the active input x but are not differentiated.

Subtypes

Constant
Cache
ConstantOrCache

source

DifferentiationInterface.Constant — Type

Constant

Concrete type of Context argument which is kept constant during differentiation.

Note that an operator can be prepared with an arbitrary value of the constant. However, same-point preparation must occur with the exact value that will be reused later.

Warning

Some backends require any Constant context to be a Number or an AbstractArray.

Example

julia> using DifferentiationInterface

julia> using ForwardDiff: ForwardDiff

julia> f(x, c) = c * sum(abs2, x);

julia> gradient(f, AutoForwardDiff(), [1.0, 2.0], Constant(10))
2-element Vector{Float64}:
 20.0
 40.0

julia> gradient(f, AutoForwardDiff(), [1.0, 2.0], Constant(100))
2-element Vector{Float64}:
 200.0
 400.0

source

DifferentiationInterface.Cache — Type

Cache

Concrete type of Context argument which can be mutated with active values during differentiation.

The initial values present inside the cache do not matter.

For some backends, preparation allocates the required memory for Cache contexts with the right element type, similar to PreallocationTools.jl.

Warning

Some backends require any Cache context to be an AbstractArray, others accept nested (named) tuples of AbstractArrays.

Example

julia> using DifferentiationInterface

julia> using ForwardDiff: ForwardDiff

julia> f(x, c) = sum(copyto!(c, x));

julia> prep = prepare_gradient(f, AutoForwardDiff(), [1.0, 2.0], Cache(zeros(2)));

julia> gradient(f, prep, AutoForwardDiff(), [3.0, 4.0], Cache(zeros(2)))
2-element Vector{Float64}:
 1.0
 1.0

source

DifferentiationInterface.ConstantOrCache — Type

ConstantOrCache

Concrete type of Context argument which can contain a mixture of constants and caches, passed along to the backend without modification.

Unlike for Cache, it is up to the user to ensure that the internal storage can adapt to the required element types, for instance by using PreallocationTools.jl directly.

source

DifferentiationInterface.prepare_pushforward — Function

prepare_pushforward(f,     backend, x, tx, [contexts...]; strict=Val(true)) -> prep
prepare_pushforward(f!, y, backend, x, tx, [contexts...]; strict=Val(true)) -> prep

Create a prep object that can be given to pushforward and its variants to speed them up.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

For in-place functions, y is mutated by f! during preparation.

Warning

The preparation result prep is only reusable as long as the arguments to pushforward do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.prepare_pushforward_same_point — Function

prepare_pushforward_same_point(f,     backend, x, tx, [contexts...]; strict=Val(true)) -> prep_same
prepare_pushforward_same_point(f!, y, backend, x, tx, [contexts...]; strict=Val(true)) -> prep_same

Create a prep object that can be given to pushforward and its variants to speed them up, if they are applied at the same point x and with the same contexts.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

For in-place functions, y is mutated by f! during preparation.

Warning

The preparation result prep is only reusable as long as the arguments to pushforward do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.pushforward — Function

pushforward(f,     [prep,] backend, x, tx, [contexts...]) -> ty
pushforward(f!, y, [prep,] backend, x, tx, [contexts...]) -> ty

Compute the pushforward of the function f at point x with a tuple of tangents tx.

To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.

Tip

Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named jvp.

source

DifferentiationInterface.pushforward! — Function

pushforward!(f,     dy, [prep,] backend, x, tx, [contexts...]) -> ty
pushforward!(f!, y, dy, [prep,] backend, x, tx, [contexts...]) -> ty

Compute the pushforward of the function f at point x with a tuple of tangents tx, overwriting ty.

To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.

Tip

Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named jvp!.

source

DifferentiationInterface.value_and_pushforward — Function

value_and_pushforward(f,     [prep,] backend, x, tx, [contexts...]) -> (y, ty)
value_and_pushforward(f!, y, [prep,] backend, x, tx, [contexts...]) -> (y, ty)

Compute the value and the pushforward of the function f at point x with a tuple of tangents tx.

To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.

Tip

Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp.

Info

Required primitive for forward mode backends.

source

DifferentiationInterface.value_and_pushforward! — Function

value_and_pushforward!(f,     dy, [prep,] backend, x, tx, [contexts...]) -> (y, ty)
value_and_pushforward!(f!, y, dy, [prep,] backend, x, tx, [contexts...]) -> (y, ty)

Compute the value and the pushforward of the function f at point x with a tuple of tangents tx, overwriting ty.

To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.

Tip

Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp!.

source

DifferentiationInterface.prepare_pullback — Function

prepare_pullback(f,     backend, x, ty, [contexts...]; strict=Val(true)) -> prep
prepare_pullback(f!, y, backend, x, ty, [contexts...]; strict=Val(true)) -> prep

Create a prep object that can be given to pullback and its variants to speed them up.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

For in-place functions, y is mutated by f! during preparation.

Warning

The preparation result prep is only reusable as long as the arguments to pullback do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.prepare_pullback_same_point — Function

prepare_pullback_same_point(f,     backend, x, ty, [contexts...]; strict=Val(true)) -> prep_same
prepare_pullback_same_point(f!, y, backend, x, ty, [contexts...]; strict=Val(true)) -> prep_same

Create a prep object that can be given to pullback and its variants to speed them up, if they are applied at the same point x and with the same contexts.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

For in-place functions, y is mutated by f! during preparation.

Warning

The preparation result prep is only reusable as long as the arguments to pullback do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.pullback — Function

pullback(f,     [prep,] backend, x, ty, [contexts...]) -> tx
pullback(f!, y, [prep,] backend, x, ty, [contexts...]) -> tx

Compute the pullback of the function f at point x with a tuple of tangents ty.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp.

source

DifferentiationInterface.pullback! — Function

pullback!(f,     dx, [prep,] backend, x, ty, [contexts...]) -> tx
pullback!(f!, y, dx, [prep,] backend, x, ty, [contexts...]) -> tx

Compute the pullback of the function f at point x with a tuple of tangents ty, overwriting dx.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp!.

source

DifferentiationInterface.value_and_pullback — Function

value_and_pullback(f,     [prep,] backend, x, ty, [contexts...]) -> (y, tx)
value_and_pullback(f!, y, [prep,] backend, x, ty, [contexts...]) -> (y, tx)

Compute the value and the pullback of the function f at point x with a tuple of tangents ty.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp.

Info

Required primitive for reverse mode backends.

source

DifferentiationInterface.value_and_pullback! — Function

value_and_pullback!(f,     dx, [prep,] backend, x, ty, [contexts...]) -> (y, tx)
value_and_pullback!(f!, y, dx, [prep,] backend, x, ty, [contexts...]) -> (y, tx)

Compute the value and the pullback of the function f at point x with a tuple of tangents ty, overwriting dx.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp!.

source

DifferentiationInterface.prepare_derivative — Function

prepare_derivative(f,     backend, x, [contexts...]; strict=Val(true)) -> prep
prepare_derivative(f!, y, backend, x, [contexts...]; strict=Val(true)) -> prep

Create a prep object that can be given to derivative and its variants to speed them up.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

For in-place functions, y is mutated by f! during preparation.

Warning

The preparation result prep is only reusable as long as the arguments to derivative do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.derivative — Function

derivative(f,     [prep,] backend, x, [contexts...]) -> der
derivative(f!, y, [prep,] backend, x, [contexts...]) -> der

Compute the derivative of the function f at point x.

To improve performance via operator preparation, refer to prepare_derivative.

source

DifferentiationInterface.derivative! — Function

derivative!(f,     der, [prep,] backend, x, [contexts...]) -> der
derivative!(f!, y, der, [prep,] backend, x, [contexts...]) -> der

Compute the derivative of the function f at point x, overwriting der.

To improve performance via operator preparation, refer to prepare_derivative.

source

DifferentiationInterface.value_and_derivative — Function

value_and_derivative(f,     [prep,] backend, x, [contexts...]) -> (y, der)
value_and_derivative(f!, y, [prep,] backend, x, [contexts...]) -> (y, der)

Compute the value and the derivative of the function f at point x.

To improve performance via operator preparation, refer to prepare_derivative.

source

DifferentiationInterface.value_and_derivative! — Function

value_and_derivative!(f,     der, [prep,] backend, x, [contexts...]) -> (y, der)
value_and_derivative!(f!, y, der, [prep,] backend, x, [contexts...]) -> (y, der)

Compute the value and the derivative of the function f at point x, overwriting der.

To improve performance via operator preparation, refer to prepare_derivative.

source

DifferentiationInterface.prepare_gradient — Function

prepare_gradient(f, backend, x, [contexts...]; strict=Val(true)) -> prep

Create a prep object that can be given to gradient and its variants to speed them up.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

Warning

The preparation result prep is only reusable as long as the arguments to gradient do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.gradient — Function

gradient(f, [prep,] backend, x, [contexts...]) -> grad

Compute the gradient of the function f at point x.

To improve performance via operator preparation, refer to prepare_gradient.

source

DifferentiationInterface.gradient! — Function

gradient!(f, grad, [prep,] backend, x, [contexts...]) -> grad

Compute the gradient of the function f at point x, overwriting grad.

To improve performance via operator preparation, refer to prepare_gradient.

source

DifferentiationInterface.value_and_gradient — Function

value_and_gradient(f, [prep,] backend, x, [contexts...]) -> (y, grad)

Compute the value and the gradient of the function f at point x.

To improve performance via operator preparation, refer to prepare_gradient.

source

DifferentiationInterface.value_and_gradient! — Function

value_and_gradient!(f, grad, [prep,] backend, x, [contexts...]) -> (y, grad)

Compute the value and the gradient of the function f at point x, overwriting grad.

To improve performance via operator preparation, refer to prepare_gradient.

source

DifferentiationInterface.prepare_jacobian — Function

prepare_jacobian(f,     backend, x, [contexts...]; strict=Val(true)) -> prep
prepare_jacobian(f!, y, backend, x, [contexts...]; strict=Val(true)) -> prep

Create a prep object that can be given to jacobian and its variants to speed them up.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

For in-place functions, y is mutated by f! during preparation.

Warning

The preparation result prep is only reusable as long as the arguments to jacobian do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.jacobian — Function

jacobian(f,     [prep,] backend, x, [contexts...]) -> jac
jacobian(f!, y, [prep,] backend, x, [contexts...]) -> jac

Compute the Jacobian matrix of the function f at point x.

To improve performance via operator preparation, refer to prepare_jacobian.

source

DifferentiationInterface.jacobian! — Function

jacobian!(f,     jac, [prep,] backend, x, [contexts...]) -> jac
jacobian!(f!, y, jac, [prep,] backend, x, [contexts...]) -> jac

Compute the Jacobian matrix of the function f at point x, overwriting jac.

To improve performance via operator preparation, refer to prepare_jacobian.

source

DifferentiationInterface.value_and_jacobian — Function

value_and_jacobian(f,     [prep,] backend, x, [contexts...]) -> (y, jac)
value_and_jacobian(f!, y, [prep,] backend, x, [contexts...]) -> (y, jac)

Compute the value and the Jacobian matrix of the function f at point x.

To improve performance via operator preparation, refer to prepare_jacobian.

source

DifferentiationInterface.value_and_jacobian! — Function

value_and_jacobian!(f,     jac, [prep,] backend, x, [contexts...]) -> (y, jac)
value_and_jacobian!(f!, y, jac, [prep,] backend, x, [contexts...]) -> (y, jac)

Compute the value and the Jacobian matrix of the function f at point x, overwriting jac.

To improve performance via operator preparation, refer to prepare_jacobian.

source

DifferentiationInterface.SecondOrder — Type

SecondOrder

Combination of two backends for second-order differentiation.

Danger

SecondOrder backends do not support first-order operators.

Constructor

SecondOrder(outer_backend, inner_backend)

Fields

outer::AbstractADType: backend for the outer differentiation
inner::AbstractADType: backend for the inner differentiation

source

DifferentiationInterface.prepare_second_derivative — Function

prepare_second_derivative(f, backend, x, [contexts...]; strict=Val(true)) -> prep

Create a prep object that can be given to second_derivative and its variants to speed them up.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

Warning

The preparation result prep is only reusable as long as the arguments to second_derivative do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.second_derivative — Function

second_derivative(f, [prep,] backend, x, [contexts...]) -> der2

Compute the second derivative of the function f at point x.

To improve performance via operator preparation, refer to prepare_second_derivative.

source

DifferentiationInterface.second_derivative! — Function

second_derivative!(f, der2, [prep,] backend, x, [contexts...]) -> der2

Compute the second derivative of the function f at point x, overwriting der2.

To improve performance via operator preparation, refer to prepare_second_derivative.

source

DifferentiationInterface.value_derivative_and_second_derivative — Function

value_derivative_and_second_derivative(f, [prep,] backend, x, [contexts...]) -> (y, der, der2)

Compute the value, first derivative and second derivative of the function f at point x.

To improve performance via operator preparation, refer to prepare_second_derivative.

source

DifferentiationInterface.value_derivative_and_second_derivative! — Function

value_derivative_and_second_derivative!(f, der, der2, [prep,] backend, x, [contexts...]) -> (y, der, der2)

Compute the value, first derivative and second derivative of the function f at point x, overwriting der and der2.

To improve performance via operator preparation, refer to prepare_second_derivative.

source

DifferentiationInterface.prepare_hvp — Function

prepare_hvp(f, backend, x, tx, [contexts...]; strict=Val(true)) -> prep

Create a prep object that can be given to hvp and its variants to speed them up.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

Warning

The preparation result prep is only reusable as long as the arguments to hvp do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.prepare_hvp_same_point — Function

prepare_hvp_same_point(f, backend, x, tx, [contexts...]; strict=Val(true)) -> prep_same

Create a prep object that can be given to hvp and its variants to speed them up, if they are applied at the same point x and with the same contexts.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

Warning

The preparation result prep is only reusable as long as the arguments to hvp do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.hvp — Function

hvp(f, [prep,] backend, x, tx, [contexts...]) -> tg

Compute the Hessian-vector product of f at point x with a tuple of tangents tx.

To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.

source

DifferentiationInterface.hvp! — Function

hvp!(f, tg, [prep,] backend, x, tx, [contexts...]) -> tg

Compute the Hessian-vector product of f at point x with a tuple of tangents tx, overwriting tg.

To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.

source

DifferentiationInterface.gradient_and_hvp — Function

gradient_and_hvp(f, [prep,] backend, x, tx, [contexts...]) -> (grad, tg)

Compute the gradient and the Hessian-vector product of f at point x with a tuple of tangents tx.

To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.

source

DifferentiationInterface.gradient_and_hvp! — Function

gradient_and_hvp!(f, grad, tg, [prep,] backend, x, tx, [contexts...]) -> (grad, tg)

Compute the gradient and the Hessian-vector product of f at point x with a tuple of tangents tx, overwriting grad and tg.

To improve performance via operator preparation, refer to prepare_hvp and prepare_hvp_same_point.

source

DifferentiationInterface.prepare_hessian — Function

prepare_hessian(f, backend, x, [contexts...]; strict=Val(true)) -> prep

Create a prep object that can be given to hessian and its variants to speed them up.

Depending on the backend, this can have several effects (preallocating memory, recording an execution trace) which are transparent to the user.

Warning

The preparation result prep is only reusable as long as the arguments to hessian do not change type or size, and the function and backend themselves are not modified. Otherwise, preparation becomes invalid and you need to run it again. In some settings, invalid preparations may still give correct results (e.g. for backends that require no preparation), but this is not a semantic guarantee and should not be relied upon.

Danger

The preparation result prep is not thread-safe. Sharing it between threads may lead to unexpected behavior. If you need to run differentiation concurrently, prepare separate prep objects for each thread.

When strict=Val(true) (the default), type checking is enforced between preparation and execution (but size checking is left to the user). While your code may work for different types by setting strict=Val(false), this is not guaranteed by the API and can break without warning.

source

DifferentiationInterface.hessian — Function

hessian(f, [prep,] backend, x, [contexts...]) -> hess

Compute the Hessian matrix of the function f at point x.

To improve performance via operator preparation, refer to prepare_hessian.

source

DifferentiationInterface.hessian! — Function

hessian!(f, hess, [prep,] backend, x, [contexts...]) -> hess

Compute the Hessian matrix of the function f at point x, overwriting hess.

To improve performance via operator preparation, refer to prepare_hessian.

source

DifferentiationInterface.value_gradient_and_hessian — Function

value_gradient_and_hessian(f, [prep,] backend, x, [contexts...]) -> (y, grad, hess)

Compute the value, gradient vector and Hessian matrix of the function f at point x.

To improve performance via operator preparation, refer to prepare_hessian.

source

DifferentiationInterface.value_gradient_and_hessian! — Function

value_gradient_and_hessian!(f, grad, hess, [prep,] backend, x, [contexts...]) -> (y, grad, hess)

Compute the value, gradient vector and Hessian matrix of the function f at point x, overwriting grad and hess.

To improve performance via operator preparation, refer to prepare_hessian.

source

DifferentiationInterface.check_available — Function

check_available(backend)

Check whether backend is available (i.e. whether the extension is loaded) and return a Bool.

source

DifferentiationInterface.check_inplace — Function

check_inplace(backend)

Check whether backend supports differentiation of in-place functions and return a Bool.

source

DifferentiationInterface.outer — Function

outer(backend::SecondOrder)
outer(backend::AbstractADType)

Return the outer backend of a SecondOrder object, tasked with differentiation at the second order.

For any other backend type, this function acts like the identity.

source

DifferentiationInterface.inner — Function

inner(backend::SecondOrder)
inner(backend::AbstractADType)

Return the inner backend of a SecondOrder object, tasked with differentiation at the first order.

For any other backend type, this function acts like the identity.

source

DifferentiationInterface.DifferentiateWith — Type

DifferentiateWith

Function wrapper that enforces differentiation with a "substitute" AD backend, possible different from the "true" AD backend that is called.

For instance, suppose a function f is not differentiable with Zygote because it involves mutation, but you know that it is differentiable with Enzyme. Then f2 = DifferentiateWith(f, AutoEnzyme()) is a new function that behaves like f, except that f2 is differentiable with Zygote (thanks to a chain rule which calls Enzyme under the hood). Moreover, any larger algorithm alg that calls f2 instead of f will also be differentiable with Zygote (as long as f was the only Zygote blocker).

Tip

This is mainly relevant for package developers who want to produce differentiable code at low cost, without writing the differentiation rules themselves. If you sprinkle a few DifferentiateWith in places where some AD backends may struggle, end users can pick from a wider variety of packages to differentiate your algorithms.

Warning

DifferentiateWith only supports out-of-place functions y = f(x) without additional context arguments. It only makes these functions differentiable if the true backend is either ForwardDiff, reverse-mode Mooncake, or if it automatically importing rules from ChainRules (e.g. Zygote). Some backends are also able to manually import rules from ChainRules. For any other true backend, the differentiation behavior is not altered by DifferentiateWith (it becomes a transparent wrapper).

Warning

When using DifferentiateWith(f, AutoSomething()), the function f must not close over any active data. As of now, we cannot differentiate with respect to parameters stored inside f.

Fields

f: the function in question, with signature f(x)
backend::AbstractADType: the substitute backend to use for differentiation

Note

For the substitute AD backend to be called under the hood, its package needs to be loaded in addition to the package of the true AD backend.

Constructor

DifferentiateWith(f, backend)

Example

julia> using DifferentiationInterface

julia> using FiniteDiff: FiniteDiff
       using ForwardDiff: ForwardDiff
       using Zygote: Zygote

julia> function f(x::Vector{Float64})
           a = Vector{Float64}(undef, 1)  # type constraint breaks ForwardDiff
           a[1] = sum(abs2, x)  # mutation breaks Zygote
           return a[1]
       end;

julia> f2 = DifferentiateWith(f, AutoFiniteDiff());

julia> f([3.0, 5.0]) == f2([3.0, 5.0])
true

julia> alg(x) = 7 * f2(x);

julia> ForwardDiff.gradient(alg, [3.0, 5.0])
2-element Vector{Float64}:
 42.0
 70.0

julia> Zygote.gradient(alg, [3.0, 5.0])[1]
2-element Vector{Float64}:
 42.0
 70.0

source

DifferentiationInterface.MixedMode — Type

MixedMode

Combination of a forward and a reverse mode backend for mixed-mode sparse Jacobian computation.

Danger

MixedMode backends only support jacobian and its variants, and it should be used inside an AutoSparse wrapper.

Constructor

MixedMode(forward_backend, reverse_backend)

source

DifferentiationInterface.DenseSparsityDetector — Type

DenseSparsityDetector

Sparsity pattern detector satisfying the detection API of ADTypes.jl.

The nonzeros in a Jacobian or Hessian are detected by computing the relevant matrix with dense AD, and thresholding the entries with a given tolerance (which can be numerically inaccurate). This process can be very slow, and should only be used if its output can be exploited multiple times to compute many sparse matrices.

Danger

In general, the sparsity pattern you obtain can depend on the provided input x. If you want to reuse the pattern, make sure that it is input-agnostic.

Warning

DenseSparsityDetector functionality is now located in a package extension, please load the SparseArrays.jl standard library before you use it.

Fields

backend::AbstractADType is the dense AD backend used under the hood
atol::Float64 is the minimum magnitude of a matrix entry to be considered nonzero

Constructor

DenseSparsityDetector(backend; atol, method=:iterative)

The keyword argument method::Symbol can be either:

:iterative: compute the matrix in a sequence of matrix-vector products (memory-efficient)
:direct: compute the matrix all at once (memory-hungry but sometimes faster).

Note that the constructor is type-unstable because method ends up being a type parameter of the DenseSparsityDetector object (this is not part of the API and might change).

Examples

using ADTypes, DifferentiationInterface, SparseArrays
using ForwardDiff: ForwardDiff

detector = DenseSparsityDetector(AutoForwardDiff(); atol=1e-5, method=:direct)

ADTypes.jacobian_sparsity(diff, rand(5), detector)

# output

4×5 SparseMatrixCSC{Bool, Int64} with 8 stored entries:
 1  1  ⋅  ⋅  ⋅
 ⋅  1  1  ⋅  ⋅
 ⋅  ⋅  1  1  ⋅
 ⋅  ⋅  ⋅  1  1

Sometimes the sparsity pattern is input-dependent:

ADTypes.jacobian_sparsity(x -> [prod(x)], rand(2), detector)

# output

1×2 SparseMatrixCSC{Bool, Int64} with 2 stored entries:
 1  1

ADTypes.jacobian_sparsity(x -> [prod(x)], [0, 1], detector)

# output

1×2 SparseMatrixCSC{Bool, Int64} with 1 stored entry:
 1  ⋅

source

DifferentiationInterface.AutoForwardFromPrimitive — Type

AutoForwardFromPrimitive(backend::AbstractADType)

Wrapper which forces a given backend to act as a forward-mode backend, using only its native value_and_pushforward primitive and re-implementing the rest from scratch.

Tip

This can be useful to circumvent high-level operators when they have impractical limitations. For instance, ForwardDiff.jl's jacobian does not support GPU arrays but its pushforward does, so AutoForwardFromPrimitive(AutoForwardDiff()) has a GPU-friendly jacobian.

source

DifferentiationInterface.AutoReverseFromPrimitive — Type

AutoReverseFromPrimitive(backend::AbstractADType)

Wrapper which forces a given backend to act as a reverse-mode backend, using only its native value_and_pullback implementation and rebuilding the rest from scratch.

source

DifferentiationInterface.Prep — Type

Prep

Abstract supertype for all preparation results (outputs of prepare_operator functions).

Warning

The public API does not make any guarantees about the type parameters or field layout of Prep, the only guarantee is that this type exists.

source

Argument wrappers

First order

Pushforward

Pullback

Derivative

Gradient

Jacobian

Second order

Second derivative

Hessian-vector product

Hessian

Utilities

Backend queries

Backend switch

Sparsity tools

From primitive

Preparation type