topos
The guide
Differentiation
The engine reverse scan, the same rules recorded as more spec, seeds and VJPs, second derivatives, and forward mode as an outside reading.
Three ways to a gradient#
Topos is reverse mode over tensors: one scalar target, every input, about one forward of cost. There is no target-free gradient of a network; you always say which number you mean.
The three roads, in the order the crate recommends them:
Tape::differentiaterecords the chain rule as ordinary nodes and answersAdjoints: the target and one(wrt, gradient)pair per name. The derivative becomes spec — lower a forward-only entry overadjoints.roots(), and fusion and liveness apply to the chain rule itself.Run::recorded_gradientsbridges the result toParameters::step. This is compiled training.Run::backwardis the interpreter applying the same rules without recording: the engine reverse scan. It is the oracle the transform is proven against, bitwise, shipped forever — and the quick road in a loop.Entry::backward()is neither: a memory posture that retains what the engine scan reads, so a plan that did not record its derivative can still answerbackward.
The engine scan#
Run::backward takes a rank-0 target and walks the run's values backward. At every node it applies the opcode's derivative rule to the cotangent that arrived, producing one cotangent per operand; a value with several readers sums what comes back.
-((a + b) * c) + a * c = -12 d/da = -c + c = 0 d/db = -c = -4 d/dc = -(a + b) + a = -3
backward, every gradient. a feeds two subexpressions whose contributions cancel exactly, hence its zero.The code that produced it
use topos::Tape;
let tape: Tape<f64> = Tape::new();
let a = tape.leaf(2.0);
let b = tape.leaf(3.0);
let c = tape.leaf(4.0);
let expression = -((a + b) * c) + a * c;
let (a, b, c, expression) = (a.symbol(), b.symbol(), c.symbol(), expression.symbol());
let network = tape.into_network();
let run = network.forward(&network.parameters(), []);
let gradients = run.backward(expression);
show::lines([
format!("-((a + b) * c) + a * c = {}", run.of(expression)),
format!("d/da = -c + c = {}", gradients.of(a)),
format!("d/db = -c = {}", gradients.of(b)),
format!("d/dc = -(a + b) + a = {}", gradients.of(c)),
])The rules return cotangents; the scan adds them. That accumulation is stated once, in the engine, and never in a rule — which is why a above gets an honest zero from two contributions that cancel, rather than one of them.
The same rules, recording#
Derivative knowledge lives in exactly one place, written against the recordable vocabulary — the operations a tape can record — not against numbers. That body has two readings. Over Tensor it computes: that is the engine scan. Over Trace it records the same steps as ordinary nodes: that is differentiate. The rules cannot tell which reading they are under.
0 Parameter [] 1 Input [] 2 Input [] 3 Mul 0, 1 [] 4 Sub 3, 2 [] 5 Mul 4, 4 [] recorded by differentiate; node 20 is d loss / d w 6 Leaf [] 7 Mul 6, 4 [] 8 Mul 6, 4 [] 9 Leaf [] 10 Add 9, 7 [] 11 Add 10, 8 [] 12 Neg 11 [] 13 Leaf [] 14 Add 13, 11 [] 15 Leaf [] 16 Add 15, 12 [] 17 Mul 14, 1 [] 18 Mul 14, 0 [] 19 Leaf [] 20 Add 19, 17 [] 21 Leaf [] 22 Add 21, 18 [] tape: 23 nodes, 1 parameter, 2 inputs
w, appended as ordinary nodes below the divider.The code that produced it
use topos::Tape;
let tape: Tape<f64> = Tape::new();
let w = tape.parameter(0.0);
let x = tape.input(0.0);
let y = tape.input(0.0);
let error = w * x - y;
let loss = error * error;
let spec_nodes = tape.len();
let adjoints = tape.differentiate(loss, [w]);
let gradient = adjoints.of(w.symbol()).index();
show::spec_from(
tape.describe(),
spec_nodes,
&format!("recorded by differentiate; node {gradient} is d loss / d w"),
)The appended lines are readable spec. Each Leaf then Add pair is a gradient started at zero and one consumer's share added; error has two readers, so its gradient collects two shares. The picture makes the pattern visible.
error has two consumers, so its gradient is a Leaf holding zero plus one Add per consumer's share. The rules return cotangents; the scan adds them.The code that produced it
use topos::Tape;
let tape: Tape<f64> = Tape::new();
let w = tape.parameter(0.0);
let x = tape.input(0.0);
let y = tape.input(0.0);
let error = w * x - y;
let loss = error * error;
let spec_nodes = tape.len();
tape.differentiate(loss, [w]);
show::graph(Graph::from_nodes(tape.nodes()).classify_from(spec_nodes, "derived"))Because the gradient is spec, everything downstream applies to it unchanged: dead-node elimination drops the parts an entry does not declare, liveness frees what the chain rule no longer needs, patterns fuse across the forward and backward halves, and emission writes (parameters, batch) -> (loss, gradients...) as one function. Recorded reverse mode is the argument for building it this way.
One rule body, proven#
Two readings of one rule cannot drift, but a claim like that is one assert away from proof, so the crate asserts it. The engine scan and the recorded gradient must agree bit for bit — same seed, same accumulation order, same ancestor mask.
engine scan [[1.4721782245984396, -1.2963249092243496], [-1.7515114917848909, 1.2875976853034148], [-1.202283943495261, 1.4725472617489126]] recorded gradient [[1.4721782245984396, -1.2963249092243496], [-1.7515114917848909, 1.2875976853034148], [-1.202283943495261, 1.4725472617489126]] same bits in every element: asserted
The code that produced it
use topos::{Tape, Tensor};
let tape: Tape<f64> = Tape::new();
let weights = tape.parameter(Tensor::new([3, 2], [0.5, -0.25, 0.125, 0.75, -0.5, 0.25]));
let inputs = tape.input(Tensor::new([2, 3], [0.5, -1.0, 0.25, -0.75, 0.5, 1.25]));
let scores = inputs.matmul(weights).tanh().log_softmax(1);
let loss = (scores * scores).sum();
let adjoints = tape.differentiate(loss, [weights]);
let (weights, loss) = (weights.symbol(), loss.symbol());
let network = tape.into_network();
let parameters = network.parameters();
// The engine scan computes the gradient; the recorded nodes, run
// by the same interpreter, answer it again.
let run = network.forward(¶meters, []);
let engine = run.backward(loss).parameters(¶meters);
let recorded = run.recorded_gradients(&adjoints);
let bits = |tensor: &Tensor<f64>| -> Vec<u64> { tensor.to_vec().iter().map(|v| v.to_bits()).collect() };
assert_eq!(bits(engine.of(weights)), bits(recorded.of(weights)));
show::lines([
format!("engine scan {}", engine.of(weights)),
format!("recorded gradient {}", recorded.of(weights)),
"same bits in every element: asserted".to_string(),
])Seeds and vector-Jacobian products#
A plain gradient plants a one at a rank-0 target. Tape::vjp takes any target and an explicit seed of the same shape: the vector-Jacobian product Jᵀ seed. With a one-hot seed it picks a row of the Jacobian; with a gradient as the seed it records a second derivative.
x = [0.5, -0.25, 1.5] x^3 = [0.125, -0.015625, 3.375] seed = [1, 0, 0] J^T seed = [0.75, 0, 0] (3 x^2 in the seeded row, zero elsewhere)
vjp plants the seed instead of the ones a plain gradient uses, and picks out one row of the Jacobian.The code that produced it
use topos::{Tape, Tensor};
let tape: Tape<f64> = Tape::new();
let x = tape.parameter(Tensor::new([3], [0.5, -0.25, 1.5]));
let cubes = x * x * x;
// A gradient needs a scalar target; a VJP takes any target and an
// explicit seed of the same shape. This seed picks the first row
// of the Jacobian.
let seed = tape.leaf(Tensor::new([3], [1.0, 0.0, 0.0]));
let adjoints = tape.vjp(cubes, seed, [x]);
let (x, cubes) = (x.symbol(), cubes.symbol());
let network = tape.into_network();
let run = network.forward(&network.parameters(), []);
show::lines([
format!("x = {}", run.of(x)),
format!("x^3 = {}", run.of(cubes)),
"seed = [1, 0, 0]".to_string(),
format!("J^T seed = {} (3 x^2 in the seeded row, zero elsewhere)", run.of(adjoints.of(x))),
])Second derivatives#
The recorded gradient is made of ordinary differentiable nodes, so it is a valid input to the transform itself. A Hessian-vector product is vjp over a recorded gradient, seeded with a direction. There is no higher-order mode to switch on; it is the absence of a wall.
x = [0.5, -0.25, 1.5] gradient 3 x^2 = [0.75, 0.1875, 6.75] direction v = [0.25, 1, -0.5] H v 6 x . v = [0.75, -1.5, -4.5]
0 Parameter [3] 1 Mul 0, 0 [3] 2 Mul 1, 0 [3] 3 Sum 2 [] 4 Leaf [] 5 Broadcast 4 shape=[3] [3] 6 Leaf [3] 7 Add 6, 5 [3] 8 Mul 7, 0 [3] 9 Mul 7, 1 [3] 10 Leaf [3] 11 Add 10, 8 [3] 12 Leaf [3] 13 Add 12, 9 [3] 14 Mul 11, 0 [3] 15 Mul 11, 0 [3] 16 Add 13, 14 [3] 17 Add 16, 15 [3] the second derivative, recorded over the first 18 Leaf [3] 19 Leaf [3] 20 Add 19, 18 [3] 21 Leaf [3] 22 Add 21, 18 [3] 23 Leaf [3] 24 Add 23, 20 [3] 25 Leaf [3] 26 Add 25, 20 [3] 27 Mul 22, 0 [3] 28 Mul 22, 11 [3] 29 Leaf [3] 30 Add 29, 27 [3] 31 Leaf [3] 32 Add 31, 28 [3] 33 Mul 26, 0 [3] 34 Mul 26, 11 [3] 35 Add 30, 33 [3] 36 Add 32, 34 [3] 37 Leaf [3] 38 Add 37, 24 [3] 39 Leaf [3] 40 Add 39, 24 [3] 41 Leaf [3] 42 Add 41, 35 [3] 43 Leaf [3] 44 Add 43, 35 [3] 45 Mul 40, 1 [3] 46 Mul 40, 7 [3] 47 Leaf [3] 48 Add 47, 45 [3] 49 Leaf [3] 50 Add 49, 46 [3] 51 Mul 44, 0 [3] 52 Mul 44, 7 [3] 53 Add 48, 51 [3] 54 Add 36, 52 [3] 55 Leaf [3] 56 Add 55, 53 [3] 57 Leaf [3] 58 Add 57, 53 [3] 59 Sum 58 [] 60 Broadcast 59 shape=[] [] 61 Leaf [] 62 Add 61, 60 [] 63 Mul 50, 0 [3] 64 Mul 50, 0 [3] 65 Add 54, 63 [3] 66 Add 65, 64 [3] network: 67 nodes, 1 parameter, 0 inputs
vjp walks it like anything else, and the Hessian-vector product is one more root.The code that produced it
use topos::{Tape, Tensor};
let tape: Tape<f64> = Tape::new();
let x = tape.parameter(Tensor::new([3], [0.5, -0.25, 1.5]));
let value = (x * x * x).sum();
// The first derivative is recorded as ordinary nodes...
let first = tape.differentiate(value, [x]);
let gradient = first.of(x.symbol());
// ...so the second one is a VJP over the recorded gradient, seeded
// with a direction: the Hessian-vector product `H v`.
let second_starts = tape.len();
let direction = tape.leaf(Tensor::new([3], [0.25, 1.0, -0.5]));
let second = tape.vjp(gradient, direction, [x]);
let hvp = second.of(x.symbol());
let x = x.symbol();
let network = tape.into_network();
let run = network.forward(&network.parameters(), []);
show::panels([
(
"values",
show::lines([
format!("x = {}", run.of(x)),
format!("gradient 3 x^2 = {}", run.of(gradient)),
"direction v = [0.25, 1, -0.5]".to_string(),
format!("H v 6 x . v = {}", run.of(hvp)),
]),
),
(
"spec",
show::spec_from(network.describe(), second_starts, "the second derivative, recorded over the first"),
),
])examples/forward_mode.rs computes the same product the other way round — forward-over-reverse — and asserts the two agree bit for bit at dyadic values.
The backward pass as a curve#
A Field holds one payload per node: far too much to print and exactly the right amount to plot. The Euclidean norm of each gradient, along the tape, is the shape of a backward pass; a vanishing or exploding region is a shape, not a number to hunt for. The notebook card for a field draws exactly this.
The code that produced it
use malevich::{Line, Plot};
use topos::{Activation, Mlp, Module, Shape, Tape, Tensor, init};
let tape: Tape<f32> = Tape::new();
let mlp = Mlp::new(&tape, &[4, 8, 8, 1], Activation::Tanh, init::xavier(3));
let features: Tensor<f32> = init::normal(5, 1.0)(&Shape::new([16, 4]));
let targets: Tensor<f32> = init::normal(7, 1.0)(&Shape::new([16, 1]));
let error = mlp.express(tape.input(features)) - tape.input(targets);
let loss = (error * error).sum().symbol();
let network = tape.into_network();
// A field holds one payload per node; its norms are the shape of
// the backward pass.
let run = network.forward(&network.parameters(), []);
let gradients = run.backward(loss);
let norms: Vec<f64> = gradients
.payloads()
.iter()
.map(|payload| payload.to_vec().iter().map(|v| f64::from(*v) * f64::from(*v)).sum::<f64>().sqrt())
.collect();
show::plot(
Plot::new()
.layer(Line::y(norms).label("gradient norm"))
.title("the backward pass of a [4, 8, 8, 1] mlp, node by node")
.x_label("node")
.y_label("norm"),
)Forward mode, from outside#
Forward mode — one input nudge, every output — is not a library feature. It is dual arithmetic over a Recordable payload, replayed through the spec with Opcode::express: wrap every value as a pair of value and tangent, and ordinary multiplication already is the product rule. Over Dual<Tensor> the walk computes a directional derivative eagerly; over Dual<Trace> it records the tangent computation as more spec. Both are graded, bitwise, against the reverse scan.
That is the seam a new AD mode plugs into: not a second list of derivative rules, not a fork of the engine. forward_mode.rs is the exhibit, dual.rs does the same by plugging the element seam instead, and AD as a named reading tells what the decision opened.
Where next#
- Training — from gradients to steps, with the state in your hands.
- Entries and plans — compiled training as a forward-only plan over the adjoints.