Acceleration

Backends are opt-in cargo features. Turn one on and dispatch happens inside the payload: nothing to call, nothing to configure. Whatever a backend declines, the interpreter still runs. Features change speed, not which programs you can write.

The interpreter's bits stay the spec. A faster kernel that reorders float math is a labeled Fast run, never a silent effect of a feature flag.

The tables are a snapshot from one Apple M1 Pro. They show order-of-magnitude and which rung wins, not a promise for your chip. Rerun on yours: cargo bench, the throughput example, and the scripts in tools/.

Turn it on#

Accelerate: the backend's cargo feature is off in this build
Metal: the backend's cargo feature is off in this build
Cuda: the backend's cargo feature is off in this build
Simd: the backend's cargo feature is off in this build
Fused: ready
StableHlo: ready

use topos::Backend;

let mut lines = Vec::new();
for backend in Backend::ALL {
    match backend.status() {
        Ok(()) => lines.push(format!("{backend:?}: ready")),
        Err(reason) => lines.push(format!("{backend:?}: {reason}")),
    }
}
show::text(lines.join("\n"))

cargo run --release --example throughput
cargo run --release --features simd --example throughput
cargo run --release --features accelerate --example throughput
cargo run --release --features accelerate,metal --example throughput

There is nothing to detect. Optional loud mode, if you would rather fail than fall back:

topos::Backend::Metal.status().expect("metal backend unavailable");

For metal, status also warms the kernels so the first large product does not pay compilation.

Which feature#

Default. Pure Rust. Dense products on a slice path shaped for the auto-vectorizer; bit-identical to the naive loop.

simd. Portable CPU microkernels (matrixmultiply), single-threaded, AVX-512 / AVX2 / NEON at run time. Transposed and narrowed views run at full speed; broadcast operands decline.

accelerate. Apple's AMX/SME via BLAS, plus vForce maps and vDSP batch-norm. Zero extra crates. On Apple Silicon it is the usual winner for products.

metal. The crate's own GPU kernels, f32 only. Products often still lose to AMX, so Metal serves what BLAS declines and leads maps once buffers are large.

cuda. cuBLAS, libraries loaded at run time so a missing toolkit does not fail the build. Copy-bound; not yet measured on NVIDIA hardware.

Enabling a feature is the whole activation. Selection is per build and per task (size, precision, layout), never per call site. f64 never reaches Metal.

Measured#

Matrix products, 512- to 2048-square; maps are f32 tanh at 2M–8M elements.

The simd row is NEON on that M1; on x86 it is AVX-512 or AVX2 and should be the same order. The naive strided loop is 0.4 GFLOP/s there: the correctness anchor and the fallback for exotic layouts.

Whole training steps, same snapshot (batch 64, f32):

The wins track how much of the step is a product, not the ladder ratios: these convolutions are skinny, and window fills still run the built-in paths. Metal's flat MNIST row is the threshold working — every product sits below the bar, so the run matches the default bits. Accuracies differ across Fast builds because each backend sums in its own order; each trajectory is valid.

Broadcasts are views, not copies: a stride-0 window over the source. The default build already runs them at slice speed (bias-add 7.6 Gelem/s against 0.17 for a naive walk). The graph records broadcasts explicitly; nothing broadcasts by accident.

Fast and exact#

interpreter  15.916065
exact plan   15.916065
fast plan    15.916065

exact == interpreter: asserted, in every build
fast == exact: yes, no backend is on in this build

use topos::{Numerics, Shape, Tape, Tensor, init};

let tape: Tape<f32> = Tape::new();
let weights = tape.parameter(init::xavier(3)(&Shape::new([64, 64])));
let batch = tape.input(init::normal(5, 1.0)(&Shape::new([32, 64])));
let loss = batch.matmul(weights).tanh().sum().symbol();
let network = tape.into_network();
let parameters = network.parameters();

let interpreter = network.forward(&parameters, []);
let exact = network.entry([loss]).numerics(Numerics::Exact).lower().forward(&parameters, []);
let fast = network.entry([loss]).lower().forward(&parameters, []);
let bits = |tensor: &Tensor<f32>| tensor.scalar().to_bits();
assert_eq!(bits(interpreter.of(loss)), bits(exact.of(loss)));
show::lines([
    format!("interpreter  {}", interpreter.of(loss)),
    format!("exact plan   {}", exact.of(loss)),
    format!("fast plan    {}", fast.of(loss)),
    String::new(),
    "exact == interpreter: asserted, in every build".to_string(),
    format!(
        "fast == exact: {}",
        if bits(fast.of(loss)) == bits(exact.of(loss)) { "yes, no backend is on in this build" } else { "no: a backend summed in its own order" }
    ),
])

Results are a function of the payloads, the compiled features, the numerics posture, and the machine. Two identical runs of one binary never disagree.

Network::forward — whole-spec evaluation, the proving road — is exact by construction: its bits are the same in every build, on every platform. What Fast (an entry's default) gives up is bit-identity with that oracle: hardware reassociates sums. Exact restores those bits in the same process — a labeled choice, not a feature flag.

let exact = network.entry([loss]).numerics(Numerics::Exact).lower();
let fast = network.entry([loss]).lower();

A dependency that enables topos/accelerate enables it for the whole binary. An Exact entry still computes interpreter bits.

Which backend actually served is run-time data: wrap any region in Backend::tallied and read the per-formula tally — coverage declares may, the tally reports what did. The throughput example prints one for a full training step.

Serving#

A compiled plan is already a closed, statically shaped tensor function. Plan::emit_stablehlo writes it as text; an industrial runtime takes it from there. Nothing XLA-shaped links in-crate. See Vision.

Compile cost amortized, same snapshot:

The CNN is the argument: the same tape, eleven times faster, by handing the plan to XLA. Attention is already within a quarter of XLA because both ride BLAS. The interpreter is a spec, not a runner. Modules are parsed and checked against the plan whenever a JAX toolchain is present (TOPOS_STABLEHLO_VALIDATOR, TOPOS_STABLEHLO_EVALUATOR).

For a new element#

Override Elementary::gemm and Elementary::map to route a number type onto kernels; the defaults compute on the built-in paths. The engine never sees this. See The element is the seam.