Tutorial

We present a typical workflow with DifferentiationInterfaceTest.jl, building on the tutorial of the DifferentiationInterface.jl documentation (which we encourage you to read first).

julia> import Chairmarks
julia> using DataFrames
julia> using DifferentiationInterface, DifferentiationInterfaceTest
julia> using ForwardDiff: ForwardDiff
julia> using Zygote: Zygote

Introduction

The AD backends we want to compare are ForwardDiff.jl and Zygote.jl.

backends = [AutoForwardDiff(), AutoZygote()]

2-element Vector{ADTypes.AbstractADType}:
 AutoForwardDiff()
 AutoZygote()

To do that, we are going to take gradients of a simple function:

f(x::AbstractArray) = sum(sin, x)

f (generic function with 1 method)

Of course we know the true gradient mapping:

∇f(x::AbstractArray) = cos.(x)

∇f (generic function with 1 method)

DifferentiationInterfaceTest.jl relies with so-called Scenarios, in which you encapsulate the information needed for your test:

• the operator category (here :gradient)
• the behavior of the operator (either :in or :out of place)
• the function f
• the input x of the function f (and possible tangents or contexts)
• the reference first-order result res1 (and possible second-order result res2) of the operator
• the arguments prep_args passed during preparation

xv = rand(Float32, 3)
xm = rand(Float64, 3, 2)
scenarios = [
    Scenario{:gradient,:out}(f, xv; res1=∇f(xv)),
    Scenario{:gradient,:out}(f, xm; res1=∇f(xm)),
];

Testing

The main entry point for testing is the function test_differentiation. It has many options, but the main ingredients are the following:

julia> test_differentiation(
           backends,  # the backends you want to compare
           scenarios;  # the scenarios you defined,
           correctness=true,  # compares values against the reference
           type_stability=:none,  # checks type stability with JET.jl
           detailed=true,  # prints a detailed test set
       )Test Summary:                                                 | Pass  Total   Time
Testing correctness                                           |   88     88  10.2s
  AutoForwardDiff()                                           |   44     44   4.4s
    gradient                                                  |   44     44   4.3s
      Scenario{:gradient,:out} f : Vector{Float32} -> Float32 |   22     22   2.7s
      Scenario{:gradient,:out} f : Matrix{Float64} -> Float64 |   22     22   1.5s
  AutoZygote()                                                |   44     44   5.8s
    gradient                                                  |   44     44   5.4s
      Scenario{:gradient,:out} f : Vector{Float32} -> Float32 |   22     22   4.4s
      Scenario{:gradient,:out} f : Matrix{Float64} -> Float64 |   22     22   1.0s

Benchmarking

Once you are confident that your backends give the correct answers, you probably want to compare their performance. This is made easy by the benchmark_differentiation function, whose syntax should feel familiar:

table = benchmark_differentiation(backends, scenarios);

DifferentiationInterfaceTest.DifferentiationBenchmark{Float64}(DifferentiationBenchmarkDataRow{Float64}[DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :value_and_gradient, true, 1, 26833, 536, 5.213246268656717e-8, 3.0, 112.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :gradient, true, 1, 28231, 649, 4.279198767334361e-8, 2.0, 80.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :value_and_gradient, true, 1, 28539, 228, 1.2382894736842107e-7, 3.0, 160.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :gradient, true, 1, 33373, 213, 1.1335680751173709e-7, 2.0, 128.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :value_and_gradient, true, 1, 28221, 35, 8.138285714285714e-7, 25.0, 688.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :gradient, true, 1, 35138, 36, 6.267222222222223e-7, 23.0, 624.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :value_and_gradient, true, 1, 29172, 26, 1.0681538461538462e-6, 29.0, 1040.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :gradient, true, 1, 18058, 31, 9.010645161290323e-7, 27.0, 976.0, 0.0, 0.0)])

The resulting object is a table, which can easily be converted into a DataFrame from DataFrames.jl. Its columns correspond to the fields of DifferentiationBenchmarkDataRow.

df = DataFrame(table)