# Benchmark contract

Backplane separates three kinds of evidence:

1. **Control-plane evidence** — isolation, package verification, scheduling,
   limits, and receipt correctness.
2. **Runtime evidence** — model compatibility, KV cache, latency, throughput,
   GPU memory, and reversible hooks.
3. **Capability evidence** — whether a particular card improves a real workload
   against prompt, RAG, tool, and adapter baselines.

`backplane benchmark` currently measures category 1. It deliberately prints:

> Control-plane isolation, scheduling, and receipts only; this does not
> benchmark neural inference quality.

## Required production comparison

Every card claim should use the same model, prompts, hardware, decoding policy,
and held-out split across:

| Arm | Description |
|---|---|
| Base | Frozen model, no added capability |
| Prompt | Equivalent instruction or data inserted as tokens |
| Retrieval | Equivalent external memory retrieved into context |
| Tool | Equivalent deterministic external action |
| LoRA | Per-request conventional adapter |
| Backplane | Declared card stack |

Record task accuracy, failure cases, input tokens, output tokens, time to first
token, end-to-end latency, throughput, peak GPU memory, persistent bytes per
tenant, and rollout/rollback time.

The first production target is a 7B/14B multi-tenant support workload with
conflicting tenant policies, isolated memory, and exact tools.

