Memory for coding agents, graded by execution.
A real coding agent completes real repository tasks whose solutions depend on earlier sessions. The artifact is graded by execution: tests pass or they do not. No LLM judge is used in the primary endpoint.
Design in six decisions
the whole benchmark, compressedOfficial integrations, frozen and vendor-reviewed
Each product enters through its shipped Claude Code integration, such as a plugin, MCP server, or lifecycle hooks. The adapter and frozen configuration are reviewable before measurement.
Each product carries its own shipped integration
Every arm uses the same task, corpus, and fixture. The designated baseline is
claude_md, a curated instruction bundle. Memory arms publish their
instruction size so retrieval is not confused with generic agent coaching.
One neutral feed, read path measured
Every adapter receives the same verbatim session transcripts. What it keeps or discards is part of what is measured.
Executable endpoints only
Checkers compare the artifact with oracle inputs absent from the sandbox. A silent or incorrect session scores zero. Naive and informed references are asserted in CI.
The admission gate
A cell is admitted only when every arm proves its treatment was available and the sandbox was equivalent. Discard counts are published per arm. A timeout is an outcome, not a wiring fault, and is never retried.
Costs are end-to-end
Session and ingestion tokens, wall time, discarded cells, and negative transfer are recorded per arm. Prices are explicit for every live run, and small deltas are reported as noise.
The arms
same feed, same tasks, same gatebare is the floor. placebo separates memory from
the effect of extra context. claude_md is the practical baseline: a static
instruction file with no per-query retrieval cost. A memory product must beat it to earn its
additional complexity.
What each arm isolates, in full →
protocol carries the shared memory instruction with no memory
behind it. It measures the cost of asking an agent to use memory before retrieval begins.
fs_grep is the cheap retrieval control. recall_prefetch
retrieves in the harness with the exact task prompt, removing query formulation. Neither is
ranked as a competitor.
How they decompose the problem →
oracle_memory is not in the approved run because its
condition-blind bundles would violate the meaning of absent. It returns when its
bundles are condition-aware.
Where this actually stands
2026-09-02| piece | state |
|---|---|
| harness | runs; sandbox, admission gate, paired statistics, cost ledger, MCP preflight and bounded retry |
| tasks | 34 executable tasks, each with naive and informed references asserted in CI. Three require evidence combined across sessions |
| harm suite | five corpus conditions. Eleven tasks carry all four damage conditions, above the threshold of eight for reporting a condition as a result |
| arms | eight in official-003: controls, two product arms, and one reference track. mempalace is held on the public board until its vendor review window closes |
| corpus | the same bulk-ingested corpus was supplied to every arm before the grid. The write path is not measured |
| runs | official-003 is published on the leaderboard. Its headline is a null, and the run's late preregistration and absent advance announcement are disclosed |
| leaderboard | numbers come from generated summaries in results/<run_id>/, never hand-entered copy |
| reproducibility | checkable, not yet re-runnable. python -m scripts.verify_run --all re-derives the ledger, endpoints, and discard set with no credentials or model calls. Re-running memory arms still requires their external services and built indexes |
The dated status page in the repository carries the run-by-run table, the open blockers in the order they have to be cleared, and the command that re-derives every claim on this page: docs/STATUS.md. Every change to the instrument that moves a published number is recorded in docs/audit/.
This benchmark is built by the authors of recall, which competes in it.
That is exactly why the methodology is preregistered, the harness is open, every adapter
config is vendor-reviewable before any run, and all results are published, including the
ones recall loses. The full run's protocol is committed under
preregistration/ before a single session starts.