Benchmarks
What we have measured, how, and what the numbers do not show. Every number on this page was produced by a pinned harness and a pinned build. Where we ran another vendor's harness we changed nothing but the memory backend, so the comparison to that vendor is run-for-run. Scores from other vendors' own harnesses are shown as they reported them and are not comparable run-for-run.
Our rule, set before each run: write the projected score down first, run the full split, publish the complete ability profile whatever the overall comes to, and lead with the strongest and the weakest ability.
BEAM-100K: plain search scores the same (2026-11-15)
We had not measured what plain search over the raw conversation scores on this benchmark, and we could not find the number published anywhere. So we ran it: the same harness commit, dataset revision, answer and judge model, published prompts and 8,000-token evidence cap as the 89.1 below, with the memory backend replaced by search over the conversation's turns. Nothing is extracted, summarised or rewritten. Each turn is one document, ranked by BM25, by embedding similarity (the embedding model MindGraph uses), or by the two fused.
On BEAM-100K, plain search and MindGraph's graph memory score the same. Hybrid search is one point above us, inside the noise, and every baseline's paired difference to MindGraph has an interval that contains zero.
| BM25 | Vector | Hybrid | MindGraph | past.dev | |
|---|---|---|---|---|---|
| Overall, 400 questions | 88.6 | 89.4 | 90.3 | 89.3 | 92.1 |
| 95% interval over the 20 conversations | 86.5–90.3 | 87.8–90.9 | 88.6–91.8 | 87.5–90.9 | 90.6–93.5 |
| Paired difference to MindGraph | −0.6 (−2.5 to +1.3) | +0.1 (−1.7 to +1.9) | +1.0 (−0.7 to +2.8) | — | — |
| Scoring passes per question | 3 | 3 | 3 | 3 | 1 |
MindGraph's column is the three-pass reading of the 2026-11-08 run (89.3; its single protocol pass is the 89.1 below). past.dev's column is its own published single pass, with the interval computed from its published per-question results.
The ability profile
Strongest against search: contradiction resolution, 86.7 against 70.6–71.8, the only ability where our interval sits clear of all three baselines. Weakest against search: abstention (88.3 against 95.8–99.2) and information extraction (82.7 against 86.7–91.5), where returning the turns themselves beats returning what we extracted from them.
| Ability | BM25 | Vector | Hybrid | MindGraph | past.dev |
|---|---|---|---|---|---|
| Event ordering | 99.8 | 99.9 | 100.0 | 99.8 | 100.0 |
| Preference following | 99.2 | 98.5 | 99.4 | 97.7 | 100.0 |
| Abstention | 99.2 | 95.8 | 97.5 | 88.3 | 98.8 |
| Summarization | 98.3 | 97.8 | 97.2 | 96.9 | 99.4 |
| Temporal reasoning | 95.6 | 95.2 | 95.8 | 96.2 | 95.0 |
| Instruction following | 91.9 | 95.8 | 94.6 | 97.3 | 96.2 |
| Information extraction | 86.7 | 89.7 | 91.5 | 82.7 | 95.0 |
| Knowledge update | 79.0 | 84.0 | 87.7 | 80.0 | 89.4 |
| Contradiction resolution | 71.2 | 70.6 | 71.8 | 86.7 | 78.1 |
| Multi-session reasoning | 65.4 | 66.5 | 67.4 | 66.9 | 68.9 |
Per-ability intervals are in the published summary. With 40 questions per ability they are wide (multi-session reasoning spans about 55 to 79 for every system), and differences inside them are not evidence.
What this shows, and what it does not
- This split does not separate memory systems from search. BEAM-100K conversations run 93K–181K tokens and 188–392 turns, so 8,000 tokens of the best-ranked turns already hold most answers. The result says nothing about the 500K, 1M and 10M splits.
- Four abilities sit at 95–100 in every column (event ordering, preference following, summarization, temporal reasoning). The published answer prompt gives the reader part of the answer key for several abilities; that applies to every system alike, and it compresses the differences. The benchmark separates systems mainly on contradiction resolution, knowledge update, information extraction and multi-session reasoning.
- Our contradiction lead comes from a reader rule, not from retrieval. The evidence MindGraph renders ends with one sentence telling the reader to quote disagreeing statements and ask which is correct. Plain search carries no such line.
- These baselines are floors for search. They use the same embedding model as MindGraph and no reranker; a stronger embedding model or a reranker would likely score higher.
- The fill differs under the same cap. The baselines spend the whole 8,000 tokens on every question; MindGraph's evidence had a median of 6,287.
- The whole transcript in context was not a ceiling on the two conversations we scored that way (40 questions, one pass, indicative only): 88.6 against 92.1 for BM25 and 91.8 for hybrid on the same questions, almost all of it from multi-session counting questions, where a reader shown every turn over-counts.
What we are doing about it
Two things. On the read path, extracted items alone lose detail that the source text holds, so /retrieve/context is gaining a default that returns source passages chosen by the graph alongside the items; it is merged in mindgraph-rs behind a setting that stays off until its acceptance tests pass. On measurement, we are moving to benchmarks where search has measured room above it, and we now run the search baselines before our own system on every benchmark.
Method
- Harness, data, models. As for the 89.1 below: past.dev's open BEAM harness at
a712cd99with the one-line backend import, datasetMohammadta/BEAMsplit 100K at revision3205395e, answers and judging byopenai/gpt-5.6-lunawith reasoning off, the published prompts unchanged. The harness's own verification passed on every run. - Documents. One per conversation turn, rendered with its date, session number, turn number and speaker. Only turns at or before the question's moment are eligible; checked after the runs, none of the 28,777 turns shown post-dates its question.
- Ranking. BM25 with k1 1.5 and b 0.75, lowercase alphanumeric tokens, no stemming or stop list. Vector: cosine similarity over OpenAI
text-embedding-3-small. Hybrid: reciprocal-rank fusion of the two top-100 lists with constant 60. - Evidence. Whole turns in rank order until the 8,000-token cap, then shown in conversation order. Showing them in conversation order and with dated headers is our choice and likely helps event ordering and knowledge update.
- Scoring. Three passes of the answer and judge steps over the same evidence, the method behind MindGraph's 89.3; a question's score is the mean of its passes. Single passes ranged 88.1–89.0 for BM25, 88.9–89.7 for vector and 90.0–90.6 for hybrid.
- Intervals. 95% percentile bootstrap over the 20 conversations, 4,000 draws; differences to MindGraph are paired by question and resampled by conversation.
- Cost. $10.37 for all arms, three passes each, including the embeddings.
The summary with every pin, a per-question score table for each arm and the generated tables are kept with the adapter (published/beam-100k-baselines-2026-11-15/). Write to shan@mindgraph.cloud for a copy, including the per-question evidence and judge verdicts.
BEAM-100K: 89.1 (2026-11-08) — one reader rule
Read-path re-evaluation on the 2026-11-07 run's stored databases: the same 20 databases, harness commit, dataset revision, published prompts, answer and judge model, and recall shape. Nothing was re-ingested. What changed is the server binary: mindgraph-rs main after #136, which adds one sentence as the last line of every rendered evidence:
Reader rule: where two statements in the record disagree and no resolution is recorded, quote both as stated and ask the user which statement is correct before relying on either.
| MindGraph 2026-11-08 | MindGraph 2026-11-07 | MindGraph 2026-11-05 | past.dev, same harness | |
|---|---|---|---|---|
| Overall, 400 questions | 89.1 | 87.3 | 85.0 | 92.1 |
| 95% interval over the 20 conversations | 87.3–90.8 | 85.2–89.4 | 82.1–87.6 | — |
| Questions answered / failed | 400 / 0 | 400 / 0 | 400 / 0 | 400 / 0 |
The protocol scores each question once; the 89.1 is that single pass. Because one scoring pass moves a 40-question ability by four to five points on identical evidence (we measured it: the same configuration scored twice on the same databases on 2026-11-08), every development decision behind this run was read as the mean of three scoring passes per side, paired question by question. On that reading the change is 87.2 → 89.3, +2.0 with a 95% interval of +0.5 to +3.6, and it was pre-registered at 89.5 ± 1.0 before the run.
The ability profile
Strongest: event ordering at 99.8. Weakest: multi-session reasoning at 67.9, unchanged within noise; its misses are counting questions whose gold follows the dataset's narrative rather than the transcript, and past.dev sits at 69.0 on the same questions.
| Ability | MindGraph 2026-11-08 | 95% interval | 2026-11-07 | past.dev, same harness |
|---|---|---|---|---|
| Event ordering | 99.8 | 99.5–100.0 | 100.0 | 100.0 |
| Instruction following | 98.1 | 95.0–100.0 | 93.1 | 96.3 |
| Preference following | 96.9 | 93.1–100.0 | 97.5 | 100.0 |
| Summarization | 96.7 | 94.1–98.9 | 96.6 | 99.4 |
| Temporal reasoning | 95.0 | 89.4–98.8 | 98.1 | 95.0 |
| Abstention | 87.5 | 77.5–95.0 | 91.2 | 98.8 |
| Contradiction resolution | 86.9 | 81.9–91.6 | 70.9 | 78.1 |
| Information extraction | 83.1 | 74.3–91.1 | 81.4 | 95.0 |
| Knowledge update | 78.8 | 66.2–90.0 | 80.0 | 89.4 |
| Multi-session reasoning | 67.9 | 56.3–78.9 | 64.6 | 69.0 |
Intervals are bootstrap intervals over conversations. With 40 questions per ability they are wide; differences inside them are not evidence.
What moved, and why
- Contradiction resolution, 70.9 → 86.9 (three-pass reading +14.2, interval +10.9 to +17.2). The rubric rewards four things on a contradiction question; the one we kept losing was asking the user which statement is correct. Question-level replays showed the answer model already quoted both statements in 34 of 40 questions and simply never asked. A status line gated on a recorded contradiction edge reached 6 of 40 questions on these databases, because the other 34 have no edge; the unconditional rule reaches all of them. We say this plainly: the gain comes from an instruction carried in the evidence, not from retrieving anything new. It is the same instruction a careful reader would follow with a paper file.
- Instruction following, 93.1 → 98.1 (three-pass +3.5, interval +0.8 to +7.1). The rule also makes the reader more careful about stated constraints.
- Abstention, 91.2 → 87.5 (three-pass 88.3, −2.9, interval −9.2 to +3.3). Inside its interval, but the sign is the one we would expect from a rule that asks the reader to quote rather than decline, so it is on the watch list for the next run.
- Everything else moved inside noise: temporal reasoning 98.1 → 95.0 (three-pass 95.8 → 96.2), knowledge update 80.0 → 78.8 (three-pass 78.1 → 80.0), information extraction 81.4 → 83.1, multi-session 64.6 → 67.9.
Also measured, not shipped
Three other read-path changes were measured on the same databases with the same three-pass method and did not clear the floor, so they are not in the server: a transcript window chosen by query-term density instead of first match (+1.4 overall with the rule, interval touching zero, and a cost on abstention and information extraction); passage-grain transcript retrieval simulated on the harness side before any server code (+0.3 overall, abstention −5.4); and a write-time batch that folds duplicate claims and sweeps supersession and contradiction edges after ingestion (87.1 on a fresh ingest, within noise of 87.3). The sweep's edges did make the pair rendering reach 34 of 40 contradiction questions on those databases (contradiction +5.0, interval +2.4 to +7.6), which is how the rule was found.
Method
As on 2026-11-07 (below), with these differences.
- Build.
mindgraph-rs42683dc(the pull-request head; identical tree to mainf8a8c11) withmnesticd60da15d, release profile, binary SHA-25669ffde6e…. The hosted API carries it from the next deploy. - Recall. Unchanged: one
POST /retrieve/contextper question, 40 nodes, 5 transcript chunks, the question's moment asas_of,format: "text"under the 8,000-token budget; evidence tokens p50 6,287, 9 of 400 renderings truncated; recall p50 376 ms on a laptop. - Scoring. One pass per protocol (the 89.1); two further passes of the answer and judge steps on the identical evidence for the three-pass reading (89.3). The 2026-11-07 baseline was read the same way (87.2 over three passes).
- Cost and time. About $1 for the protocol pass (answers and judging) and $1.80 for the two extra passes; 207 s of scoring on four local servers. No ingestion.
- Field on this harness at 100K. past.dev 92.1. Hindsight is listed at 86.2 on past.dev's leaderboard (attributed to agentmemorybenchmark.ai) and at 73.4–75.0 on Hindsight's own blog and benchmarks site; we show both rather than choose. Mastra 83.3 on its own setup.
The run summary with all pins, the harness's own summary, a per-question score table and the three-pass score lists are kept with the adapter (published/beam-100k-2026-11-08/). Write to shan@mindgraph.cloud for the full per-question transcript.
BEAM-100K: 87.3 (2026-11-07)
Second run on past.dev's open harness, same protocol as the 2026-11-05 run below: same dataset revision, same answer and judge model, same published prompts, same scoring, same extraction models, a fresh database per conversation. What changed is MindGraph. The read path now admits evidence by when it was stated, dates every item, keeps superseded values and marks them, renders contradiction pairs as pairs, and renders the evidence itself under the harness's 8,000-token budget; ingestion now reviews each new claim against earlier statements of the same proposition and records which value replaced which. The page records both runs because the second one tests the diagnosis we published with the first.
| MindGraph 2026-11-07 | MindGraph 2026-11-05 | past.dev, same harness | |
|---|---|---|---|
| Overall, 400 questions | 87.3 | 85.0 | 92.1 |
| 95% interval over the 20 conversations | 85.2–89.4 | 82.1–87.6 | — |
| Questions answered / failed | 400 / 0 | 400 / 0 | 400 / 0 |
Paired against the 2026-11-05 run question by question, the overall gain is +2.4 (95% interval +0.1 to +4.8). Before trusting any ability's movement we measured the instrument's noise: one configuration scored twice on identical databases moved 0.1 overall and up to 4.4 points on a single 40-question ability. Knowledge update (+18.1) and contradiction resolution (+4.4) clear that floor; every decline below is inside it.
The ability profile
Strongest: event ordering at 100. Weakest: multi-session reasoning at 64.6, which did not move, because its misses are counting questions that need a different extraction change.
| Ability | MindGraph 2026-11-07 | 95% interval | 2026-11-05 | past.dev, same harness |
|---|---|---|---|---|
| Event ordering | 100.0 | 100.0–100.0 | 99.8 | 100.0 |
| Temporal reasoning | 98.1 | 96.2–100.0 | 95.0 | 95.0 |
| Preference following | 97.5 | 93.8–100.0 | 96.3 | 100.0 |
| Summarization | 96.6 | 92.8–99.1 | 97.2 | 99.4 |
| Instruction following | 93.1 | 85.6–99.4 | 95.6 | 96.3 |
| Abstention | 91.2 | 82.5–98.8 | 92.5 | 98.8 |
| Information extraction | 81.4 | 72.3–89.9 | 79.4 | 95.0 |
| Knowledge update | 80.0 | 67.5–92.5 | 61.9 | 89.4 |
| Contradiction resolution | 70.9 | 66.6–75.0 | 66.6 | 78.1 |
| Multi-session reasoning | 64.6 | 51.6–76.6 | 65.4 | 69.0 |
Intervals are bootstrap intervals over conversations. With 40 questions per ability they are wide; differences inside them are not evidence.
What moved, and why
- Knowledge update, 61.9 → 80.0. The 2026-11-05 diagnosis was that the updated value was usually in our evidence but undated and unmarked, and the answer model picked the older one. Two changes closed most of that gap, measured separately: dated, chronologically rendered evidence with per-source excerpts took it to 68.8 on the old databases (retrieval-only re-scoring, four configurations, about a dollar each), and write-time supersession took it to 80.0 on a fresh ingest: 275 replacement links and 78 contradiction links were recorded during ingestion of the 20 conversations, at no measurable extraction cost ($37.48 against $37.71).
- Contradiction resolution, 66.6 → 70.9. Both sides of a recorded contradiction now render together, dated, with the status "unresolved". The rubric item that rewards asking the user which statement is correct remains a ceiling shared by every user of this harness.
- Temporal reasoning, 95.0 → 98.1; information extraction, 79.4 → 81.4; preference following, 96.3 → 97.5. Dated items and verbatim excerpts. Information extraction reached 85.6 on the old databases under retrieval-only re-scoring; the fresh-ingest figure is inside the interval of both.
- Multi-session reasoning, 65.4 → 64.6. Unchanged. The misses are counting questions; the graph still enumerates mentions rather than distinct items. That is the next extraction change, not a retrieval one.
- Abstention, 92.5 → 91.2; instruction following, 95.6 → 93.1. One and two questions respectively; inside the measured noise.
Method
As on 2026-11-05 (below), with these differences.
- Build.
mindgraph-rscommitca57a899withmnesticd60da15d, release profile, binary SHA-256c5fe4825…. This is the server behind the hosted API from the same week. - Recall. One
POST /retrieve/contextper question with 40 nodes, 5 transcript chunks, the question's moment asas_of, andformat: "text"so the server renders the evidence under the 8,000-token budget; the adapter passes the text through unchanged. Evidence averaged 6,179 tokens; 7 of 484 recalls were truncated. Recall p50 365 ms, p95 1.46 s against a local server. - Scoring. The harness's scoring phase crashed twice on a network interruption in its own client after ingestion had completed; the intact databases were scored once more through the same recall, answer and judge steps. The published numbers are that single scoring pass.
- Cost and time. $38.42 total: $37.48 extraction, $0.76 answers, $0.18 judging. About 3 h of ingestion, 25 min of scoring.
- Field on this harness at 100K. past.dev 92.1. Hindsight is listed at 86.2 on past.dev's leaderboard (attributed to agentmemorybenchmark.ai) and at 73.4–75.0 on Hindsight's own blog and benchmarks site; we show both rather than choose. Mastra 83.3 on its own setup.
The run summary with all pins, the harness's own summary, a per-question score table and the ingestion counters are kept with the adapter (published/beam-100k-2026-11-07/). Write to shan@mindgraph.cloud for the full per-question evidence and judge verdicts.
BEAM-100K: 85.0 (2026-11-05) — the first run
BEAM (ICLR 2026) tests long-conversation memory across ten abilities. The 100K split is 20 conversations of roughly 100,000 tokens each and 400 questions: two per ability per conversation. Each answer is graded by a model judge against a rubric; event ordering is scored by rank correlation.
We ran it on past.dev's open harness with MindGraph as the memory backend and nothing else changed: same dataset revision, same answer and judge model, same published prompts, same scoring. past.dev's own result on this harness is 92.08.
| MindGraph | past.dev, same harness | |
|---|---|---|
| Overall, 400 questions | 85.0 | 92.1 |
| 95% interval over the 20 conversations | 82.1–87.6 | — |
| Questions answered / failed | 400 / 0 | 400 / 0 |
Mastra reports 83.3 on BEAM-100K for its Observational Memory on its own setup (self-reported, average of two runs). Other vendors' 100K figures circulate only in third-party tables, so we do not list them.
The ability profile
Strongest: event ordering at 99.8. Weakest: knowledge update at 61.9. The band is the range we wrote down before the run; two abilities landed below their band and six above it.
| Ability | MindGraph | 95% interval | past.dev, same harness | Our pre-registered band |
|---|---|---|---|---|
| Event ordering | 99.8 | 99.5–100 | 100.0 | 60–90 |
| Summarization | 97.2 | 95.6–98.7 | 99.4 | 55–85 |
| Preference following | 96.3 | 92.5–100 | 100.0 | 75–95 |
| Instruction following | 95.6 | 89.4–100 | 96.3 | 70–95 |
| Temporal reasoning | 95.0 | 88.7–98.8 | 95.0 | 80–95 |
| Abstention | 92.5 | 82.5–100 | 98.8 | 40–75 |
| Information extraction | 79.4 | 67.4–89.8 | 95.0 | 80–92 |
| Contradiction resolution | 66.6 | 61.6–70.9 | 78.1 | 40–75 |
| Multi-session reasoning | 65.4 | 53.2–76.7 | 69.0 | 50–70 |
| Knowledge update | 61.9 | 45.0–77.5 | 89.4 | 65–85 |
Intervals are bootstrap intervals over conversations (the unit the benchmark samples), not over questions. With 40 questions per ability they are wide; differences inside them are not evidence.
Where the points went
We read every question that scored below 0.75 and, for knowledge update, re-queried the stored graphs. The pattern is specific enough to act on.
- Knowledge update. All 15 misses answered the older value. In 11 of them the updated fact was in the evidence we returned, often near the top. Our evidence items carry a session label but no per-item date, and the fact that superseded the old one was never marked as doing so, so the answer model had nothing to order by. In one case the graph held 14 near-duplicate nodes of the old value against one of the new.
- Multi-session reasoning. 13 of 19 weak answers were counting questions ("how many features did we discuss"). The graph enumerates every mention and the answer over-counts.
- Information extraction. Questions asking for something as it was first stated lose to our summaries. Transcript chunks carry the original wording, but they render last and the harness's 8,000-token evidence cap cut them on 376 of 400 recalls.
- Contradiction resolution. All 40 answers found the conflict. The rubric also rewards asking the user which statement is correct; the harness's answer prompt instructs the model not to settle the question, so that item is a ceiling shared by everyone using this harness. About nine misses were ours: the second statement did not surface.
- Abstention lost three questions to answers assembled from entity summaries. The remaining abilities lost one or two questions each.
What this changed (written 2026-11-05; the 2026-11-07 run above tests it)
The three weakest abilities point at the same place: time on the read path. Every evidence item will carry its date and the context will be returned in chronological order under as_of; supersession will mark the earlier value in the evidence itself; duplicate folding will collapse same-attribute restatements. Counting questions get aggregate nodes. The renderer will reserve budget for transcript chunks before relations. We will re-run the same split on the same harness after those land and publish both numbers.
Method
- Harness.
pastdotdev/benchmarksat commita712cd99(harness version 2), one line changed sorun.pyimports the memory API from a backend module. DatasetMohammadta/BEAM, split 100K, revision3205395e, the revision past.dev's published result used. - Answer and judge model.
openai/gpt-5.6-lunathrough OpenRouter, reasoning off, exactly as the harness ships. Prompts are ExaBase's published prompts, verified by SHA. Answer and judge are not MindGraph's models; they are the benchmark's. - Recall. One
POST /retrieve/contextper question: 27 nodes (9 by graph expansion), 3 articles, 5 transcript chunks, with the question's moment asas_ofso knowledge that first appears after that moment is excluded. Rendered to text and capped at the harness's 8,000 tokens. - Ingestion. One MindGraph server with a fresh database per conversation. Sessions ingested oldest first with their dates and participants. Extraction on our production model mix (Claude Haiku 5.5 on four passes, Qwen 3.6 Plus on the epistemic pass, GPT-5.5 as fallback, reasoning off); embeddings
text-embedding-3-small. - Build.
mindgraph-rscommit7ac29d60withmnesticd60da15d, release profile, binary SHA-256 recorded in the summary. - Cost and time. $38.67 total: $37.71 extraction (about $1.89 per conversation), $0.79 answers, $0.16 judging; embeddings not counted. 3 h 55 min wall clock, almost all ingestion. Recall p50 360 ms, p95 1.4 s against a local server.
- Identity. BEAM's user identity is a past.dev concept; MindGraph registers no reader. The user's name reaches extraction as a session participant.
- Pilot. The first two conversations ran first as a gate (score 0.842, $3.09); the full run resumed from them without re-running.
The run summary with all pins, the harness's own summary, and a per-question score table are kept with the adapter. Write to shan@mindgraph.cloud for a copy, including the full per-question evidence and judge verdicts.
Retrieval latency
Measured p50 and p95 for the retrieval endpoints over the last 30 days, from production telemetry, with the method on the page: mindgraph.cloud/status/latency.
In progress, not yet numbers
- LoCoMo. One-conversation diagnostic only (80.3 on categories 1–4 with the graph, 71.1 with transcript chunks alone, same ingest). The full run waits on extraction fixes; the published key has known errors and we will say so when we publish.
- Retrieval precision. A judge-labelled relevance set over our own corpus, with an owner audit of the judge (agreement 0.794). It drives ranking decisions internally and is not a comparable benchmark.
- LongMemEval-S. Dry run only; the full 500-question run is deferred on cost.