One neutral feed in.
One executed artifact out.

An arm is not a contestant. Each one exists to remove a competing explanation for what the product does, so that a difference at the end has one story left that fits it. Read down the table and the design is the argument: by the last row, "the memory layer did it" is the only surviving account.

A task is a fixture repository, a task spec, and an executable checker. Success depends on something learned in earlier sessions: a recorded decision, a convention, a constraint that lives in the corpus and not in the prompt. The suite holds 34 tasks (identifiers like ts-tz-utc, ts-stable-sort, ts-log-mask). Three of them, the xs- set, state their governing rule across two sessions rather than one, so no single document answers them: a suite where combining sessions is never necessary cannot detect a product that combines them.

Every task ships two reference solutions, asserted in CI on every commit:

The experience feed is verbatim recorded agent session transcripts, pinned by a sha256 manifest. It is deliberately not a curated fact list: it carries the noise, the dead ends and the distractor sessions real agent history carries. No arm gets a different feed, and the only asymmetry a product can gain is what it extracts.

It is 4,900 documents per condition, and it was made harder on purpose. The 196-document feed every earlier run used was saturated: hit@10 was 1.000, so every memory arm found the governing session every time and the grid could not separate "the product retrieved badly" from "the agent never searched". On the current corpus bm25 hit@1 is 0.182 against 0.485, and voyage hit@10 is 0.879 against 1.000. Containment is checked against the built corpus rather than trusted: no fact term of any task appears in any synthetic document, or the absent condition would be silently broken for that task.

That change breaks comparability. No number measured on this corpus may be differenced against a number from any earlier run.

Every task described above places its governing fact in the corpus, so that suite can only ask whether memory helps. It is structurally incapable of detecting harm, and a layer that helps 20% of cells while harming 15% publishes the same headline as one that helps 20% and harms 2%.

So a second suite varies what the corpus contains, never what a product does about it. Every arm ingests identical bytes under each condition. Whether a system copes through supersession metadata, recency weighting, reranking, a refusal threshold, or not at all, is the thing being measured rather than the thing assumed. Damage is defined against the memory-free arm: failing a cell that bare solved.

Silent failure is the standing hazard of agent benchmarks: an MCP server that never attached, a hook that never fired, a sandbox missing its files. A session in that state measures nothing, and scoring it poisons the average in whichever direction luck chooses.

So a grid cell is discarded, not scored, unless every arm proves its treatment was applied:

The primary endpoint is task success: the checker passes or it does not. Analysis is paired per task, arms are contrasted against the claude_md baseline, and deltas below the preregistered minimum effect are reported as noise rather than dressed up as findings.

Costs are end-to-end. Ingestion tokens, session tokens, wall time and negative-transfer counts land in one per-arm ledger beside the success rate, because a layer that buys two points for triple the tokens is a different product than its headline suggests. No arm has been run at a matched budget, so the ledger carries success per million tokens and reports the asymmetry rather than smoothing it.

Two details that sound like bookkeeping and are not. Input is not one price: fresh input, cache reads and cache creation are metered separately, because one arm's input can be two-thirds cache reads while a baseline's is under half, and a single rate then overstates spend unevenly between exactly the two arms being compared. Prices are stated, never defaulted: a live run refuses to start without them. Compare runs on tokens.

An arm that ingests with a model on the benchmark host reports zero hosted tokens and names the model, so its zero is never read as zero cost.

Task success says whether the memory path worked, not which part failed. A memory arm can lose three ways: the agent never searched, it searched badly, or the store did not hold the answer. recall_prefetch separates the first from the rest by running retrieval in the harness with the exact task prompt and injecting what comes back. It is in the official run, as an upper bound; it is never ranked against the products, because an arm that cannot lose does not belong in a ranking.

oracle_memory, which injects the exact evidence and removes retrieval altogether, is not in the run. Its bundles are keyed by task and carry no corpus condition, so under absent it would supply an answer that condition is defined not to contain. It returns when its bundles are condition-aware. The contrasts below are the decomposition it belongs to, stated so its absence is a recorded choice.

Most memory benchmarks ask a model questions about synthetic conversations and score the answers with another model: a needle someone planted, found under a judge whose failure modes correlate with the thing being judged. Five properties here are chosen against that.

The grade is execution. A checker runs the artifact against oracle inputs the sandbox never contained. No partial credit, no rubric, no judge in the primary endpoint, so a fluent wrong answer scores what a silent one does: zero.

The corpus is raw. Verbatim recorded sessions, dead ends and noise included. Distilling signal from that stream is what a memory product claims to do, so the benchmark refuses to do it for anyone. The only asymmetry available is what a product keeps.

Harm is measured. A suite where every governing fact sits in the corpus can only ask whether memory helps, and a layer that helps 20% of cells while harming 15% publishes the same headline as one that helps 20% and harms 2%. The corpus conditions ask whether the product notices stale, contradictory and off-subsystem evidence.

The comparison is paired. Every arm meets the same task, seed and fixture, so a cell is a within-subject comparison rather than a difference of averages across arms that met different work. Arm order is randomised per cell and recorded.

Treatment is proved. A stdio server that fails to start is invisible in a transcript: no memory tool calls, which looks exactly like a model that chose not to search. The gate discards any cell where an arm cannot prove its treatment applied.

The result is a benchmark that can return a null and survive it. If memory layers do not beat a good static instruction file on real coding work, that is the finding, and it is published as readily as the opposite.

Every measured run is preregistered: the question, the predictions, the endpoints, the contrast families, the exclusion rules and the sizing are committed before the first session starts. The run scripts enforce the mechanical half: they refuse to start while the preregistration directory is dirty.