How six memory products work, how each was run here, where it helped,
where it hurt, and what it cost. The leaderboard says who scored what; this page is my reading
of why.
written 2026-09-26 by Giulio D'Ermedata: official-003 and four joined runsmodel deepseek/deepseek-v4-flash
Read this first
I build RE-call, one of the six products compared here. That is why
every run is preregistered, every adapter is open, and every loss is published. It is also
why you should weigh my notes on RE-call, and on its competitors, with that in mind.
The prose on this page is written by hand and dated. The figures in each
product's strip are generated from the same run artifacts as the leaderboard, so the two can
be checked against each other. Where the prose quotes a number, it is as of the date above.
01
What holds across the field
six findings
No memory product clearly beats a static instruction file.
Five of the six intervals cross zero, so none of those five is distinguishable from
claude_md. The one interval that excludes zero, supermemory's, lies below it.
The controls make the point sharper: bare, with no memory at all, and
placebo, with inert prose, score as high as the best product. On this
corpus, extra text in the context window moves success about as much as a memory layer
does.
Memory helps most when a fact is missing or stale.
The gains concentrate in absent and superseded. RE-call
gains 17 and 13 points there, Claude Mem 11 and 7, Graphiti 13 on absent, cognee 9 on
superseded. Those are the conditions in which a product can add something the static
file lacks: a check that nothing governs the task, or the newer of two rules.
Memory hurts when facts conflict.
On contradictory, no product beats claude_md; Claude Mem
ties it and the other five fall below it, mempalace by 12 points. A retrieval layer tends
to surface one side of a conflict with confidence, and the agent acts on it. None of the
products tested turned a detected conflict into a question for the user often enough to
show.
More context is not more success.
Agent tokens per task run from about twice the baseline (Graphiti, cognee) to nearly
thirteen times (Claude Mem), and the ordering by cost does not follow the ordering by
success. The expensive part is usually not the search itself but what the search leaves
in the context window, re-read on every later turn.
The cost of building the store is invisible for most products.
Only cognee's ingest was metered in tokens: its graph extraction read the whole corpus
once per condition, which makes it the largest bill on the board. Graphiti's and Claude
Mem's ingest ran inside the product and was never metered; mempalace and supermemory
embedded on local models the meter cannot see. Anyone comparing cost has to add a write
side bill that this benchmark cannot price for four of the six products. The leaderboard
states each one's status beside its agent cost for that reason.
Whether the agent searches matters as much as what the store holds.
Across admitted sessions, the agent called memory in about 93% of Claude Mem's (with a
guard forcing the first search), 67% of mempalace's, 56% of RE-call's, 30% of cognee's
and 16% of Graphiti's. Supermemory has no search tool; its hooks inject context whether
or not the task needs it. A store the agent does not query cannot help, and a store that
answers every prompt unasked can hurt.
02
Product by product
in leaderboard order
RE-call
MCP server · official-003 · preregistration 026
How it works
RE-call stores chunks with their embeddings in Postgres and serves them through an MCP
server. An index is built as a generation, and a generation is served only after a
calibration run has certified a relevance threshold for it. Every hit comes back with a
verdict: usable, superseded (with a pointer to the document that replaced it), or low
confidence; when nothing clears the threshold the search abstains instead of returning the
nearest text.
How it was run here
Eight read tools, including evidence and reasoning tools; write tools withheld. Voyage
embeddings, strict trust mode, reranker off. Nothing is injected: the agent decides whether
to search. No model rewrites the corpus at ingest.
How it performed
It is the highest scoring product and leads in present, absent and superseded, which is
what the verdicts are for: an abstention on absent and a superseded label on a
stale rule are exactly the signals those conditions reward. It gains nothing on
contradictory or adjacent. A conflict comes back as two confident
hits, and a wrong scope fact comes back as a good match; neither is flagged. In an earlier
run the reasoning tools meant to handle conflicts were never called at all. The lead over
claude_md is not established: the interval crosses zero.
Pros and cons
Claude Mem
hooks and MCP search · claude-mem-official-019 · preregistration 079
How it works
A Claude Code plugin. Lifecycle hooks capture what happens in a session as observations,
a background worker writes them to SQLite immediately and backfills a vector index, and a
search server exposes them in steps: search for candidates, read a timeline around one, then
fetch the full observations. The design is progressive disclosure, so the agent pays for
detail only when it asks for it.
How it was run here
The corpus was imported as raw transcripts rather than compressed by the plugin's
observer model, so the store held verbatim sessions, not the plugin's own summaries. The
first-search guard was switched on: it blocks every other tool until the agent has searched
once. It was added after an unguarded session ignored the search instruction, and the result
is labelled as guarded for that reason. Claude Mem was never sent a review invitation.
How it performed
Second overall, ahead of claude_md in four conditions and level with it on
contradictory, the only product not to lose there. The guard is doing real work: the agent
searched in nearly every session. The price is context. It used the most agent tokens per
task of any arm, and most of them are the same context read again on each turn. Its ingest
was not metered, so the real bill is higher still.
Pros and cons
cognee
MCP server · cognee-001 · preregistration 027
How it works
cognee turns documents into a knowledge graph. Text is added, then a cognify step has a
language model extract entities and the relations between them into a graph database, while
chunks are embedded into a vector index. A search runs over both, and a result may be an
answer synthesised from the graph rather than a quoted passage.
How it was run here
One read tool, recall; the write and forget tools withheld. Local fastembed
embeddings, with graph extraction on the benchmark model through a hosted API. The shared
corpus was ingested into a base store and copied per condition.
How it performed
The best product on adjacent and second on superseded, both
conditions where structure should help: a relation carries the subsystem it belongs to. It
lost ground on absent, where a synthesised answer can read as a finding when
there is none. Two things limit the result. The agent searched in fewer than a third of
sessions, well under what its preregistration expected. And it has the most discarded cells
of any product, almost all of them sessions in which its MCP server never connected. On the
agent side it is cheap; counted with its ingest, it is the most expensive product measured.
Pros and cons
Graphiti
MCP server · official-007-graphiti · preregistration 084
How it works
Graphiti is an open source temporal knowledge graph. Each ingested episode is read by a
language model that extracts entities and the facts linking them, and, as the project
documents it, each fact carries the time it held, so a newer fact can retire an older one
rather than sit beside it. Search runs over facts, entities and the episodes they came
from.
How it was run here
The official MCP server with six read tools, extraction and embeddings through hosted
models, ingest by the controller only. The run deviated from its preregistration in six
ways, published with the run's provenance. One difference matters more than any of them:
Graphiti was given the standard corpus without the distractor sessions. Its store held
between 120 and 143 sessions per condition, all of the corpus it was offered, while every
product it is ranked against searched about 4,900, most of them distractors.
How it performed
Close to the middle of the field, strong on absent, and the cheapest
product per task in agent tokens. None of that is comparable with the other products. On
the corpus Graphiti searched, retrieval is close to saturated: in this benchmark's own
measurement, a strong embedder puts the right session in its first ten hits every time. Its search task was easier than
theirs, and some of its low cost comes from having less to search. That it still gained
little over claude_md on an easier corpus is itself informative, and the agent
searched in only about one session in six. The finding I most wanted from this product, a
gain on superseded from its handling of time, did not appear. It needs a rerun
on the full corpus before I would draw a conclusion about Graphiti from it.
Pros and cons
mempalace
MCP server · official-003 · preregistration 026
How it works
mempalace mines conversations into drawers inside a palace, organised by wing and room,
and keeps them verbatim. It runs locally: a vector store with a small local embedding model,
searched by a mix of vector similarity and keyword match, plus navigation tools that walk
the palace's structure.
How it was run here
The published MCP server with twenty read and navigation tools; tools that write were
withheld, and the agent's attempts to call them were refused. The corpus was mined natively
from the transcripts, with no hosted model anywhere in the path.
How it performed
Second among products on present, where a clean fact is waiting, and the
largest loss on contradictory apart from supermemory. That pairing is what
verbatim storage predicts: the original words are there to be found, and so are both sides
of a conflict, with nothing to rank one above the other. Its agent cost is among the higher
ones, and its ingest cost is time on the host rather than tokens.
supermemory is a memory service with a Claude Code plugin. Its hooks inject a user
profile when a session starts and relevant memories on every prompt, so the agent never
decides whether to look: memory arrives with the task. Normally a language model extracts
memories from what it is given, asynchronously.
How it was run here
The self-hosted build, with local embeddings and a local model. Its own extraction did
not finish within the ingest window, so the corpus went in as raw transcript chunks stored
as static memories. That makes this a measurement of the plugin's injection on raw memories,
not of supermemory's full pipeline, and I state it as the largest caveat on this row.
How it performed
The only product clearly worse than claude_md, and the collapse is worst
on superseded and adjacent. The streams show why. Every session
opened with a large injected block, and in the one I read closely the persistent profile
was a raw transcript from an unrelated project. Sessions were short: far fewer output tokens
and far less time than any other arm, and many ended before the agent wrote the file the
task asked for. Injection without relevance filtering turned memory into noise the agent
could not opt out of.
Pros and cons
03
What this analysis cannot tell you
read before quoting it
The write path is not measured. Every product was handed the same corpus
before the grid and never wrote to its own store, so a product whose value lies in what it
extracts and consolidates while it works gets no credit for that here.
One model, one agent. Every session ran Claude Code on
deepseek/deepseek-v4-flash. A stronger model may use the same memory
differently.
Joined runs. Four products were measured in their own runs,
on different dates, and are compared with official-003 on the cells both share.
Their differences are indicative, not paired to the last cell.
Graphiti searched a different corpus. Its run used the
corpus without the distractor sessions, so its row answers an easier question than the
other five and is not ranked against them on equal terms.
Cost is list price. Agent tokens are priced at one published
rate for every arm, with cached input charged as ordinary input. A real bill with cache
discounts would narrow the gap for the products that re-read the most context.
The protocol, every adapter and every run are in the
repository; the method is on
the method page.