Tutorial

We present a typical workflow with DifferentiationInterfaceTest.jl, building on the tutorial of the DifferentiationInterface.jl documentation (which we encourage you to read first).

julia> import Chairmarks
julia> using DataFrames
julia> using DifferentiationInterface, DifferentiationInterfaceTest
julia> using ForwardDiff: ForwardDiff
julia> using Zygote: Zygote

Introduction

The AD backends we want to compare are ForwardDiff.jl and Zygote.jl.

backends = [AutoForwardDiff(), AutoZygote()]

2-element Vector{ADTypes.AbstractADType}:
 AutoForwardDiff()
 AutoZygote()

To do that, we are going to take gradients of a simple function:

f(x::AbstractArray) = sum(sin, x)

f (generic function with 1 method)

Of course we know the true gradient mapping:

∇f(x::AbstractArray) = cos.(x)

∇f (generic function with 1 method)

DifferentiationInterfaceTest.jl relies with so-called Scenarios, in which you encapsulate the information needed for your test:

• the operator category (here :gradient)
• the behavior of the operator (either :in or :out of place)
• the function f
• the input x of the function f (and possible tangents or contexts)
• the reference first-order result res1 (and possible second-order result res2) of the operator
• the arguments prep_args passed during preparation

xv = rand(Float32, 3)
xm = rand(Float64, 3, 2)
scenarios = [
    Scenario{:gradient,:out}(f, xv; res1=∇f(xv)),
    Scenario{:gradient,:out}(f, xm; res1=∇f(xm)),
];

Testing

The main entry point for testing is the function test_differentiation. It has many options, but the main ingredients are the following:

julia> test_differentiation(
           backends,  # the backends you want to compare
           scenarios;  # the scenarios you defined,
           correctness=true,  # compares values against the reference
           type_stability=:none,  # checks type stability with JET.jl
           detailed=true,  # prints a detailed test set
       )Test Summary:                                                 | Pass  Total   Time
Testing correctness                                           |   88     88  10.8s
  AutoForwardDiff()                                           |   44     44   4.6s
    gradient                                                  |   44     44   4.5s
      Scenario{:gradient,:out} f : Vector{Float32} -> Float32 |   22     22   2.8s
      Scenario{:gradient,:out} f : Matrix{Float64} -> Float64 |   22     22   1.6s
  AutoZygote()                                                |   44     44   6.2s
    gradient                                                  |   44     44   6.1s
      Scenario{:gradient,:out} f : Vector{Float32} -> Float32 |   22     22   5.0s
      Scenario{:gradient,:out} f : Matrix{Float64} -> Float64 |   22     22   1.1s

Benchmarking

Once you are confident that your backends give the correct answers, you probably want to compare their performance. This is made easy by the benchmark_differentiation function, whose syntax should feel familiar:

table = benchmark_differentiation(backends, scenarios);

DifferentiationInterfaceTest.DifferentiationBenchmark{Float64}(DifferentiationBenchmarkDataRow{Float64}[DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :value_and_gradient, true, 1, 25280, 525, 5.114285714285715e-8, 3.0, 112.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :gradient, true, 1, 19531, 501, 4.351497005988024e-8, 2.0, 80.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :value_and_gradient, true, 1, 25059, 222, 1.2414864864864865e-7, 3.0, 160.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :gradient, true, 1, 35352, 204, 1.0735784313725491e-7, 2.0, 128.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :value_and_gradient, true, 1, 56617, 15, 8.396e-7, 25.0, 688.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :gradient, true, 1, 31573, 36, 6.442500000000001e-7, 23.0, 624.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :value_and_gradient, true, 1, 27869, 25, 1.0944e-6, 29.0, 1040.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :gradient, true, 1, 29014, 30, 9.083333333333334e-7, 27.0, 976.0, 0.0, 0.0)])

The resulting object is a table, which can easily be converted into a DataFrame from DataFrames.jl. Its columns correspond to the fields of DifferentiationBenchmarkDataRow.

df = DataFrame(table)