Ranked by what ran.
Task success on executable checkers, paired by task against the
claude_md baseline. Numbers are generated from published run artifacts.
Memory products compared
ranked by task successEach product against the claude_md baseline, a curated
static instruction file. The 95% interval is the range the true difference from
claude_md most likely lies in: when it runs from a negative to a positive number,
the product cannot be told apart from the baseline. An interval in bold excludes zero. The pros and
cons under each product are my reading of its run; the full reasoning is on the analysis
page.
| # | product | success | vs claude_md | 95% interval | cells | agent tokens / task | ingest |
|---|
Cells scored are admitted task, seed and condition cells; discarded cells failed the admission gate and are counted, not scored. A product measured in its own run is compared on the cells it shares with official-003.
Reading the cost. Agent tokens per task are what the
coding agent itself read and wrote in one session, metered identically for every arm; the
multiple is against claude_md, and the dollar figure prices those tokens at one
rate for every arm (the run's list price). Building each
product's store is a separate, one-time cost, shown under ingest as what it is: metered in
tokens, not metered (the product called its own model and the bill is missing, not zero), run
on a local model the token meter cannot see, or not in the ledger at all. Products whose
ingest is unmetered are cheaper here than they are.
This benchmark tests failure as well as recall. The corpus is empty, stale, contradictory, or out of scope in 65% of the scored cells. A clean governing fact is present in 35%.
The table measures useful memory under realistic liability, not retrieval on a clean prompt.
Where each product helps and where it costs
task success by corpus conditionpresent holds a clean governing fact. The other four test
whether memory hurts: absent has no answer to find, superseded holds a
stale one, contradictory holds a conflict, and adjacent holds a
plausible fact from the wrong scope. Each condition is ranked on its own, best first, with
claude_md placed where it lands; the small number is the difference from it.
Percentages are solved cells over admitted cells. The baseline row is over all of official-003's admitted cells; a joined product was scored on the subset it shares with them, so its differences are indicative.
Controls and reference tracks
not products, never rankedThese arms price the grid rather than compete in it. bare
and placebo scoring above most products is the finding, not a footnote: on this
corpus, extra plausible text helps about as much as a memory layer, which is why no product is
declared a winner.
| arm | what it isolates | task success | vs claude_md | 95% interval |
|---|
Reference tracks diagnose the memory path: they run in the grid and are never ranked. See the method page.
Every note the automatic analysis generated
What makes a result official
or it does not appear hereA run appears on this page only when all of the following hold, in this order:
| gate | requirement |
|---|---|
| 1 · preregistered | protocol committed under preregistration/ before the first session; the harness refuses to start while that directory is dirty |
| 2 · announced | the run is announced before it happens, not after it succeeds |
| 3 · gated | every scored cell passed the admission gate; discard counts published per arm |
| 4 · published in full | per-session logs, streams, admission verdicts and costs land in results/<run_id>/, wins and losses alike |
The results/ tree holds the evidence behind this page.
The dated project status and verification commands are in
docs/STATUS.md.