Tutorial

We present a typical workflow with DifferentiationInterfaceTest.jl, building on the tutorial of the DifferentiationInterface.jl documentation (which we encourage you to read first).

julia> import Chairmarks
julia> using DataFrames
julia> using DifferentiationInterface, DifferentiationInterfaceTest
julia> using ForwardDiff: ForwardDiff
julia> using Zygote: Zygote

Introduction

The AD backends we want to compare are ForwardDiff.jl and Zygote.jl.

backends = [AutoForwardDiff(), AutoZygote()]

2-element Vector{ADTypes.AbstractADType}:
 AutoForwardDiff()
 AutoZygote()

To do that, we are going to take gradients of a simple function:

f(x::AbstractArray) = sum(sin, x)

f (generic function with 1 method)

Of course we know the true gradient mapping:

∇f(x::AbstractArray) = cos.(x)

∇f (generic function with 1 method)

DifferentiationInterfaceTest.jl relies with so-called Scenarios, in which you encapsulate the information needed for your test:

• the operator category (here :gradient)
• the behavior of the operator (either :in or :out of place)
• the function f
• the input x of the function f (and possible tangents or contexts)
• the reference first-order result res1 (and possible second-order result res2) of the operator
• the arguments prep_args passed during preparation

xv = rand(Float32, 3)
xm = rand(Float64, 3, 2)
scenarios = [
    Scenario{:gradient,:out}(f, xv; res1=∇f(xv)),
    Scenario{:gradient,:out}(f, xm; res1=∇f(xm)),
];

Testing

The main entry point for testing is the function test_differentiation. It has many options, but the main ingredients are the following:

julia> test_differentiation(
           backends,  # the backends you want to compare
           scenarios;  # the scenarios you defined,
           correctness=true,  # compares values against the reference
           type_stability=:none,  # checks type stability with JET.jl
           detailed=true,  # prints a detailed test set
       )Test Summary:                                                 | Pass  Total  Time
Testing correctness                                           |   88     88  9.8s
  AutoForwardDiff()                                           |   44     44  4.4s
    gradient                                                  |   44     44  4.4s
      Scenario{:gradient,:out} f : Vector{Float32} -> Float32 |   22     22  2.8s
      Scenario{:gradient,:out} f : Matrix{Float64} -> Float64 |   22     22  1.4s
  AutoZygote()                                                |   44     44  5.3s
    gradient                                                  |   44     44  5.3s
      Scenario{:gradient,:out} f : Vector{Float32} -> Float32 |   22     22  4.3s
      Scenario{:gradient,:out} f : Matrix{Float64} -> Float64 |   22     22  1.0s

Benchmarking

Once you are confident that your backends give the correct answers, you probably want to compare their performance. This is made easy by the benchmark_differentiation function, whose syntax should feel familiar:

table = benchmark_differentiation(backends, scenarios);

DifferentiationInterfaceTest.DifferentiationBenchmark{Float64}(DifferentiationBenchmarkDataRow{Float64}[DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :value_and_gradient, true, 1, 26548, 574, 4.859233449477353e-8, 3.0, 112.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :gradient, true, 1, 38354, 487, 4.1554414784394254e-8, 2.0, 80.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :value_and_gradient, true, 1, 28453, 226, 1.2558849557522126e-7, 3.0, 160.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoForwardDiff(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :gradient, true, 1, 29123, 244, 1.1582786885245902e-7, 2.0, 128.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :value_and_gradient, true, 1, 28811, 35, 8.083714285714286e-7, 25.0, 688.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Vector{Float32} -> Float32, :gradient, true, 1, 19063, 45, 6.209111111111112e-7, 23.0, 624.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :value_and_gradient, true, 1, 26960, 26, 1.0750769230769232e-6, 29.0, 1040.0, 0.0, 0.0), DifferentiationBenchmarkDataRow{Float64}(AutoZygote(), Scenario{:gradient,:out} f : Matrix{Float64} -> Float64, :gradient, true, 1, 29280, 31, 9.081290322580646e-7, 27.0, 976.0, 0.0, 0.0)])

The resulting object is a table, which can easily be converted into a DataFrame from DataFrames.jl. Its columns correspond to the fields of DifferentiationBenchmarkDataRow.

df = DataFrame(table)