Why it is shaped this way

AD as a named reading

Two usual walks over one graph, and a third one written outside the crate.

You recorded a loss. You want derivatives. Automatic differentiation walks the graph and applies the chain rule. It does not rewrite the math by hand, and it does not poke the inputs with a small ε.

Two usual walks. Reverse mode: one output, every input. That is training. Forward mode: one input nudge, every output. That is cheap when there are few inputs, and it is how you build second derivatives.

Most stacks treat those walks as engine loops. Here they are two ways of reading the same graph.

The decision#

The tape is written once. After that you only read it: run it, differentiate it, schedule it, print it.

The local derivative of each operation lives in one place, written against the operations a tape can record, not against numbers. That body already has two readings. One computes. The other records the same steps as ordinary nodes. The rules cannot tell which reading they are under.

That is recorded reverse mode. It was taken so training could compile.

What it opened#

x                 = [0.5, -0.25, 1.5]
gradient  3 x^2   = [0.75, 0.1875, 6.75]
direction v       = [0.25, 1, -0.5]
H v       6 x . v = [0.75, -1.5, -4.5]

   0  Parameter                         [3]
   1  Mul            0, 0               [3]
   2  Mul            1, 0               [3]
   3  Sum            2                  []
   4  Leaf                              []
   5  Broadcast      4  shape=[3]       [3]
   6  Leaf                              [3]
   7  Add            6, 5               [3]
   8  Mul            7, 0               [3]
   9  Mul            7, 1               [3]
  10  Leaf                              [3]
  11  Add            10, 8              [3]
  12  Leaf                              [3]
  13  Add            12, 9              [3]
  14  Mul            11, 0              [3]
  15  Mul            11, 0              [3]
  16  Add            13, 14             [3]
  17  Add            16, 15             [3]
the second derivative, recorded over the first
  18  Leaf                              [3]
  19  Leaf                              [3]
  20  Add            19, 18             [3]
  21  Leaf                              [3]
  22  Add            21, 18             [3]
  23  Leaf                              [3]
  24  Add            23, 20             [3]
  25  Leaf                              [3]
  26  Add            25, 20             [3]
  27  Mul            22, 0              [3]
  28  Mul            22, 11             [3]
  29  Leaf                              [3]
  30  Add            29, 27             [3]
  31  Leaf                              [3]
  32  Add            31, 28             [3]
  33  Mul            26, 0              [3]
  34  Mul            26, 11             [3]
  35  Add            30, 33             [3]
  36  Add            32, 34             [3]
  37  Leaf                              [3]
  38  Add            37, 24             [3]
  39  Leaf                              [3]
  40  Add            39, 24             [3]
  41  Leaf                              [3]
  42  Add            41, 35             [3]
  43  Leaf                              [3]
  44  Add            43, 35             [3]
  45  Mul            40, 1              [3]
  46  Mul            40, 7              [3]
  47  Leaf                              [3]
  48  Add            47, 45             [3]
  49  Leaf                              [3]
  50  Add            49, 46             [3]
  51  Mul            44, 0              [3]
  52  Mul            44, 7              [3]
  53  Add            48, 51             [3]
  54  Add            36, 52             [3]
  55  Leaf                              [3]
  56  Add            55, 53             [3]
  57  Leaf                              [3]
  58  Add            57, 53             [3]
  59  Sum            58                 []
  60  Broadcast      59  shape=[]       []
  61  Leaf                              []
  62  Add            61, 60             []
  63  Mul            50, 0              [3]
  64  Mul            50, 0              [3]
  65  Add            54, 63             [3]
  66  Add            65, 64             [3]
network: 67 nodes, 1 parameter, 0 inputs

Plate 1. A second derivative is reverse mode of a gradient. The gradient was recorded as ordinary nodes, so vjp walks it like anything else, and the Hessian-vector product is one more root.

The gradient graph is just more graph. You can print it, compile it, emit it, and differentiate it again. A second derivative is not a new feature. It is reverse mode of a gradient, with an explicit seed.

A new AD mode is then the same kind of thing, not a new engine. Walk the spec. Speak the same operations. Forward mode is the exhibit. You do not write a new derivative for every opcode. You wrap each value as a pair — the value, and how it moves if an input nudges — and ordinary multiply already is the product rule. Replay the spec over those pairs. The directional derivative falls out.

Reverse-mode knowledge stays where it is. Forward-mode knowledge is that pair arithmetic. The slow, obvious run is still the check. A new reading has to match it, bit for bit. The idea does not need a fork.

Spelled today#

You can print every node (Opcode, Node, describe). Opcode::express runs one instruction over any payload that speaks the recordable operations. Opcode::vjp is the public name of the reverse-mode rule. Over tensors, a walk is the interpreter. Over Trace, it records again. Tape::differentiate / Tape::vjp are reverse mode as extra graph. Run::backward is the check.

Forward mode lives in examples/forward_mode.rs: a Dual payload, not a crate type. Computing the pairs and recording the pairs match each other, and both match reverse mode.

This section may rot; the rest must not.

Not this#

A second list of derivative rules for forward mode. The pair arithmetic is the forward-mode body.
Dual numbers as a built-in. The opening is the seam, not a new type in the crate.
Treating Entry::backward() as a compiler. It keeps buffers so the engine scan can still run. That is a memory choice.
Letting a dependent register a new instruction. New primitives land in the crate, under what earns an instruction.

See Vision for named readings, Recorded reverse mode for the one rule body, and The element is the seam for the other plug — a new number, not a new walk.