agent-memory-benchAMB / AMB
Submit & reproduce

Same feed. Same gate.
Your write path.

Any memory product with a published Claude Code integration can enter. The eight rules below are the ones every existing arm plays by, and the harness enforces most of them mechanically.

This page describes the scientific product leaderboard, not the prize challenge. The challenge is still in design and will use a private held out task and oracle pack. The public repository is suitable for smoke tests and audit, but it cannot be the blind scoring surface. See the challenge contract for the readiness gates.

01

The rules

enforced by the harness where possible

  1. Enter through your published integration

    A product competes as its own shipped Claude Code integration: a plugin, an MCP server, or lifecycle hooks. No bespoke benchmark builds, no unreleased branches. If your users cannot install it, the benchmark does not run it.

  2. One adapter directory, hash-pinned

    Everything an arm is lives in adapters/<name>/: the adapter implementing the MemoryAdapter contract, version pins, and config.frozen.json whose hash is recorded before the run. A config change after the freeze means a new run, not an amended one.

  3. Ingest the neutral feed as it is

    Every adapter receives identical bytes: verbatim recorded session transcripts, pinned by a sha256 manifest. Your extraction pipeline decides what to keep, and that decision is part of what is measured. No task-specific tuning, no peeking at the task suite.

  4. Prove the treatment or lose the cell

    The admission gate discards any cell where the arm cannot prove it was actually applied: MCP tools listed at session init, hooks fired with output, sandbox files digest-verified, no cross-arm contamination. Discard counts are published per arm.

  5. Real credentials or an honest refusal

    Arm credentials come from .env, never from the repository. An absent key makes the harness refuse the arm at startup rather than fake it. Local arms run via docker compose, with their extraction LLM traffic metered through the harness proxy so ingestion tokens are counted.

  6. Vendor review before the run

    Every vendor is publicly invited to review their adapter and frozen config before any measured run. The invitation, the response, or the documented silence is committed in adapters/<name>/VENDOR_REVIEW.md. Silence does not block the run; it is simply on the record.

  7. Preregistration binds the run

    The question, predictions, endpoints, exclusion rules and sizing are committed under preregistration/ before the first session. The run scripts refuse to start while that directory is dirty. Numbers in a committed preregistration are never edited; corrections are appended.

  8. Results publish in full, win or lose

    Per-session logs, streams, admission verdicts and the complete cost ledger land in results/<run_id>/. There is no private preview and no retraction path: a preregistered run that embarrasses an arm, including the authors' own, is published like any other.

02

Adding your product

a pull request, not a form

Submission is a pull request against the repository. It should contain, and review will check for, exactly four things:

filecontents
adapters/<name>/adapter.py implements the MemoryAdapter contract in harness/adapters/base.py: ingest the feed, install the integration, tear down cleanly
adapters/<name>/config.frozen.json the exact configuration the run uses, hash-pinned; defaults your users would get, not a tuned special
adapters/<name>/VENDOR_REVIEW.md from the template; records the review invitation and its outcome
adapters/<name>/pins package and image versions, exact; the run must be reconstructible from them
03

Re-run and test

trust nothing, execute everything

The harness, the tasks, the checkers and both reference solutions are in the open repository. Verifying the benchmark's own claims takes one command:

# clone, then: harness self-tests, task validation,
# and the CI assertion that every naive reference fails
# and every informed reference passes
python -m pytest tests/ -q

A real measured run additionally needs:

requirementwhy
Claude Code CLI ≥ 2.1.221 below that, a pending MCP server runs the session without its tools while reporting success; the admission gate exists because that happened
.env credentials per-arm keys from .env.example; a missing key refuses that arm rather than faking it
clean preregistration/ the run scripts refuse to start while the preregistration directory is dirty
explicit prices --price-in, --price-out and --price-as-of are required for a live run and have no defaults anywhere, because three runners once carried three different ones and none matched the frozen rates. Dry runs need none
docker for the self-hosted arms. docker/compose.yaml brings up the vector database and the harness image, and starts no memory server, so one-command full reproduction is not there yet
Reproduction is the product

If a published result cannot be regenerated from the pinned versions, the frozen configs, the sha256-pinned corpus and the committed preregistration, that is a bug in the benchmark and should be filed as one. Disagreement with a result starts with results/<run_id>/, which contains every session's logs, admission verdicts and costs.

By that standard the benchmark half fails its own rule, and says so here rather than in a footnote. The product arm is now pinned to a released package and its frozen config names environment variables instead of one machine's paths, so it no longer has to be edited to run elsewhere. What is still missing: the published runs resolved that package from a local checkout, there is no versions.lock, and the compose stack starts a database but no memory server, so a reader must still supply Postgres, an embedding key and a built index. Tracked in docs/STATUS.md.