Context that holds up.
Results you can inspect.
Better recall. The right code. More completed work. Four studies test what project context makes possible—with the methods, comparisons, and measured losses open to inspection.
10,000 free credits · No card required
Explore the evidenceAnswer accuracy · official GPT-4o judge
agentmemory sits within our 95% confidence interval. Competitor scores are vendor-published, not re-run.
Explore the comparisonResults as of October 27, 2026. ContextStream scored 96.8% (484 of 500) on the full LongMemEval-S suite with the benchmark's official judge, Wilson 95% CI [94.9%, 98.0%]: level with the highest published score (agentmemory's self-reported 96.2%), and above Mastra's 94.87% and supermemory's 85.4%. Read by Claude Opus 4.6, the reader most published results use, the same recall scores 94.0% on the tuning half. On LoCoMo it scored 95.7% (1474 of 1540). In production code search, the best answer file is in the first 10 results 98.77% of the time. In agent task trials, 23 of 24 tasks passed with ContextStream versus 14 of 24 without.
Last updated
Full LongMemEval-S result.
LongMemEval-S tests conversational memory over roughly 115k-token multi-session haystacks. We ran the complete 500-instance suite through the production API, with a disclosed reader and the official pinned GPT-4o judge, in one run registered before it was read.
484 of 500 correct (Claude Opus 5.5 reader, routed from the question text). Wilson 95% CI [94.9%, 98.0%]. On the 249 questions held out from tuning the score is 96.0% (239/249, CI [92.8%, 97.8%]), which is the figure to trust most. On the same official judge this is above Mastra's 94.87% and supermemory's 85.4%, and level with agentmemory's self-reported 96.2%, the highest published score we know of, which sits inside our interval; we say level with, not ahead of. The reader chooses its strategy from the question text alone, never from the dataset's question-type label. Zep, ByteRover and Mem0 report on other or unstated judges, so they are a different comparability class (see note).
Correct
Complete LongMemEval-S run, not a subset: 245/251 on the half used to choose the setup, 239/249 on a held-out half.
Held-out half
239 of 249 questions never used for tuning, against a bar of 237 recorded before the run. The unbiased estimate of this setup.
Evidence retrieved
Every answer-bearing session reached the reader for 491 of 500 questions; at least one did for all 500.
Wilson 95% CI
Above Mastra's 94.87% and supermemory's 85.4% on the same judge; agentmemory's 96.2% sits inside the interval (level).
| System | LongMemEval-S | Source | Scope |
|---|---|---|---|
| ContextStream | 96.8% | This run | Full 500 · official GPT-4o judge · Claude Opus 5.5 reader · 96.0% on the held-out half |
| agentmemory | 96.2% | agentmemory README | Self-reported single run · GPT-4o judge · Claude Opus 4.6 reader · its write-up also loads the easier oracle file |
| Mastra | 94.87% | Mastra research | Self-reported · GPT-4o judge · gpt-5-mini reader |
| Mem0 | 94.4% | Mem0 blog | Self-reported · reader and judge not stated |
| ByteRover | 92.8% | ByteRover blog | Vendor-published · Gemini 3.x judge, not the official GPT-4o autoeval |
| Zep | 90.2% | Zep research | Vendor-published · gpt-5.4 reader and gpt-5.4 judge, not the official GPT-4o autoeval |
| supermemory | 85.4% | supermemory comparison | Vendor-published · GPT-4o judge · cited 2026-06-13 |
Competitor numbers are cited from each vendor's own publication and were not re-run by ContextStream. On judge comparability: LongMemEval's official autoeval is gpt-4o-2024-08-06, which ContextStream uses, as do agentmemory, Mastra and supermemory. Zep scores its 90.2% with gpt-5.4 as both reader and judge, ByteRover its 92.8% with a Gemini 3.x judge, and Mem0 does not state its judge, so those three are not directly comparable to the GPT-4o-judged numbers here; a different judge can move scores in either direction. agentmemory's figure is a single self-reported run whose write-up also loads LongMemEval's easier oracle file. Readers differ across all rows: with the same recall, Claude Opus 4.6 (agentmemory's reader) scores 94.0% on our tuning half.
Multi-session counting is still our weakest family: every answer-bearing session reached the reader for 96.2% of those questions, and 92.5% were answered correctly. Of its 10 misses, 7 had every answer-bearing session in hand: most of what remains is counting across sessions, not recall.
These results come from project context your repo already produces.
Full LoCoMo result.
LoCoMo asks 1,540 questions about ten long conversations between two people: single facts, facts joined across sessions, dates, and inferences. We ran every published question through the production API, graded by the judge EverMemOS published its comparison with, and registered the run before it was read.
1474 of 1540 correct (Claude Opus 5.5 reader). Wilson 95% CI [94.6%, 96.6%]. On the 770 questions held out from tuning the score is 95.8% (738/770, CI [94.2%, 97.0%]). An independent audit found 99 of LoCoMo's official answers wrong, which caps an honest score on the original key at 93.6%; set those questions aside and the score is 97.8%. On the same judge this is above every system in the EverMemOS paper's comparison, EverMemOS's own 93.05% included.
Correct
Categories 1–4, every published question: 736/770 on the half used to choose the setup, 738/770 on a held-out half.
Held-out half
738 of 770 questions never used for tuning, against a bar of 721 recorded before the run.
Audited answer key
1409/1441 once the 99 questions whose official answer an independent audit found wrong are set aside.
Wilson 95% CI
Above every system in the EverMemOS paper's same-judge table, its own 93.05% included.
| System | LoCoMo | Source | Scope |
|---|---|---|---|
| ContextStream | 95.7% | This run | All 1,540 · Claude Opus 5.5 reader · 95.8% on the held-out half |
| EverMemOS | 93.05% | EverMemOS paper | Self-reported · GPT-4.1-mini reader |
| Zep | 85.22% | EverMemOS paper | Run by EverMemOS · GPT-4.1-mini reader |
| MemOS | 80.76% | EverMemOS paper | Run by EverMemOS · GPT-4.1-mini reader |
| MemU | 66.67% | EverMemOS paper | Run by EverMemOS · GPT-4.1-mini reader |
| Mem0 | 64.20% | EverMemOS paper | Run by EverMemOS · GPT-4.1-mini reader |
The other rows come from the EverMemOS paper (arXiv 2601.02163, Table 1), which ran each system with GPT-4.1-mini answering and the same judge prompts; it averages GPT-4o-mini with two auxiliary judges, where we take the majority of three GPT-4o-mini runs. Those numbers were produced by EverMemOS, not by each vendor, and we did not re-run anyone. Other setups: Zep self-reports 94.7% (gpt-5.4 reader and gpt-5.4 judge); Mem0 self-reports 92.5% (reader and judge not stated). Different readers and judges, so we show them without ranking; Zep's 94.7% sits inside our interval. Our reader is newer than theirs: Claude Opus 4.6 reading the same recall scores 93.8% (722/770) on the tuning half.
100.00% useful-file recall. 98.77% best-answer recall.
We mined verbatim developer queries from merged pull requests, ran each published lane twice cold and twice warm, and had two model families grade equal-depth pooled results against immutable source. A third model saw only their disagreements. GitHub's rate-limited REST search is reported separately on its fixed requested subset.
Queries judged
Every query received an independent validity judgment; invalid queries remain in the published sensitivity view.
Valid queries
162 had a judged positive for ranked retrieval; 12 hard negatives and three verified absences test whether search can return nothing.
Useful file in the top 10
ContextStream auto in production on October 27, 2026, four-cell mean across two cold and two warm runs, with zero request errors. August 1: 99.38%.
Best-answer file in the top 10
The stricter primary-answer measure, on the same independently judged positive queries. August 1: 93.83%.
Best-answer Recall@10 for ContextStream auto in production, on the same 177 judged-valid queries and scored against the frozen August 1 judgments. It follows the August 1 comparison below; it is not a re-run of it.
First-useful rank
MRR. August 1: 0.949.
Overall order
nDCG@10. August 1: 0.765.
Search time p50 / p95
Measured on the server, four-cell means.
End-to-end p50 / p95
The client ran in a different region from the one serving the benchmark workspace, so end-to-end latency includes a cross-region round trip that the August 1 measurement did not. Server-side search time is reported beside it.
Results the August judges never saw count as misses until the new rankings are pooled and judged; they were 13.5% of auto's top-10 results. Auto rejected 100.00% of the 12 hard negatives but only 33.33% of the three verified absences, down from two of three on August 1: *.r and Arg::help returned results. Auto failed the benchmark gate auto.successful_latency_p50_ms.maximum_each: 227 ms against a 125 ms ceiling. Evidence and per-cell values: the follow-up result.
Useful-file Recall@10 on the same 177-query judged-valid corpus. This was a separate, first-attempt exact-SHA production follow-up, so it is not silently mixed into the August 1 industry comparison below.
Best-answer Recall@10
A highest-relevance answer file appeared in the first ten results.
First-useful rank
MRR rewards putting the first useful file as close to rank one as possible.
p50 / p95
Four-cell means across two cold and two warm production executions.
Frozen gates passed
Source coverage was 3930/3930 (100%); no favorable rerun was performed.
The Guided follow-up is publishable for this exact deployed change; it does not by itself establish a new industry ranking. Verified-absence rejection remained 66.67%: Arg::help still returned a plausible false positive. That measured loss remains part of the result.
August 1 comparison · 2026-08-01
| System | Queries | Recall@10 | Primary@10 | First-useful rank | Overall order | Verified-absence rejection | Successful p50 | Failed requests |
|---|---|---|---|---|---|---|---|---|
| ContextStream auto no local checkout | 177 | 99.38% | 93.83% | 0.949 | 0.765 | 66.67% | 95 ms | 0 / 708 |
| ContextStream refactor no local checkout | 177 | 99.38% | 93.83% | 0.946 | 0.736 | 100.00% | 135 ms | 0 / 708 |
| Embedding-only (BGE-small) local checkout | 177 | 85.80% | 72.84% | 0.689 | 0.484 | 0.00% | 12 ms | 0 / 708 |
| ripgrep fixed literal local checkout | 177 | 68.52% | 42.59% | 0.657 | 0.366 | 66.67% | 13 ms | 0 / 708 |
| GitHub REST code search no local checkout | 29 | 0.00% | 0.00% | 0.000 | 0.000 | N/A | 494 ms | 72 / 116 |
Quality values are four-cell means across two cold and two warm runs. Latency is calculated only over successful responses, while every error or missing execution remains a quality miss and is reported separately. “First-useful rank” is MRR: it rewards putting the first useful file earlier. “Overall order” is nDCG@10: it rewards placing all higher-relevance files nearer the top. Verified-absence rejection tests three independently confirmed queries whose answer is not in the pinned repository; auto correctly abstained on two and returned one plausible false positive. ContextStream auto rejected the 12 hard negatives separately at 100%. GitHub covers its disclosed fixed subset, so its row is not a full-corpus head-to-head. Scope details: ContextStream auto: Full judged-valid corpus; production API. ContextStream refactor: Full judged-valid corpus; specialized production API lane. Embedding-only (BGE-small): Full judged-valid corpus; local checkout. ripgrep fixed literal: Full judged-valid corpus; local checkout. GitHub REST code search: Fixed 30-query requested subset; moving default branches and unsupported/corpus-mismatch responses remain misses. Procedural evidence only, not a quality head-to-head.
This is a fixed-pool, model-judged benchmark—not human qrels. New systems or changed rankings must be pooled and reviewed again. The complete evidence pack, reproduction commands, agreement statistics, variance, and negative results are in the published results file.
Three tasks needed knowledge the repository did not have.
A release procedure. A customer-data spec. An ops-handoff format. Real teams carry this kind of knowledge in decisions, runbooks, chats, and previous agent sessions. The benchmark asks whether that knowledge can reach the next agent before work starts.
Nine attempts across the three context-dependent tasks failed. The agents could only use what the repository exposed, and the missing knowledge was not there.
Every run with context connected pulled the relevant runbook, decision, or skill before acting.
These results come from project context your repo already produces.
Knowledge arrives before the work starts.
A session with project context does not have to rediscover what the team already learned. Relevant lessons, prior diagnoses, and conventions appear in the first context call.
The agent searches and re-derives first. It may still find the fix, but it spends turns rediscovering a failure mode the team had already captured.
The relevant prior diagnosis arrives at the start of the run. The agent can spend its budget on the work instead of repeating the investigation.
Same agent. Same repo. Same tasks. Context was the variable.
Each run started from a fresh clone and a fresh session. The prompts stayed identical. Hidden acceptance tests checked the work after the fact, and a blind judge scored the diffs without seeing which setup produced them.
Tasks
The suite covers repo-solvable work and tasks that require team knowledge outside the code.
Trials
Each task ran three times per setup to catch one-off luck and failure.
Scored runs
Every run used a fresh isolated clone and session.
Agent model
Claude Sonnet 4.6 was used for both arms of the benchmark.
Across all tasks, context changed 14/24 into 23/24.
The same agent, the same repository, the same eight tasks. The only change was whether ContextStream was connected.
No ContextStream
ContextStream connected
What this does and does not claim.
| Control | How it runs | Why it matters |
|---|---|---|
| LongMemEval-S full run | 500 instances, each ingested fresh into its own project through the production API, recalled in the product's precise mode | Reports the complete benchmark with a confidence interval, not a subset, and measures the recall customers get rather than a benchmark-only pipeline. |
| Pinned judge | Official GPT-4o autoeval judge (gpt-4o-2024-08-06) with the verbatim evaluation prompts | Keeps the score comparable to vendor-published LongMemEval numbers that use the same judge. |
| Disclosed reader | Claude Opus 5.5, one answer per question, no filtering step | The reader is a disclosed, swappable component, and it matters: with the same recall mode, Claude Opus 4.6, the reader most published results use, scores 94.0% (236/251) on the tuning half, against Opus 5.5's 97.6%. |
| Label-free reader routing | The reader picks its prompt and strategy from the question's wording alone | A live memory system never sees LongMemEval's question-type label. Our rounds before October 23, including the 90.0% published in June, routed on that label, which flatters the result by about 3 points on the held-out half. This result does not. |
| Pre-registered holdout | Configuration and bar committed before any held-out question was retrieved; one run, one read | The setup was chosen on one half of the questions (97.6%) and confirmed on the other (96.0%) against a bar of 237/249 recorded beforehand. Our plan set a stricter 240/249 for claiming to be above the leaders; the run scored 239, so we say level with the top published score, not ahead of it. |
| LoCoMo protocol | Categories 1–4 (1,540 questions), one project per conversation, the EverMemOS framework's judge prompts on gpt-4o-mini, three runs, majority vote | The judge with which EverMemOS published LoCoMo scores for itself, Zep, MemOS, MemU and Mem0. Category 5 is left out, as in the published results we cite: its official scoring is broken. Pre-registered the same way as LongMemEval, with a bar of 721/770 for the held-out half. |
| Audited answer key | Every LoCoMo score is also given without the 99 questions an independent audit found wrong | With 6.4% of the official answers wrong, 93.6% is the best score attainable on the original key; a lenient judge credits some answers to those questions, so scores above it are partly leniency. The audited figure removes that effect. |
| Competitor handling | Cite vendors' own published numbers, with source and date | We never re-run a competitor. Avoids the credibility trap of re-implementing a rival incorrectly and posting a deflated number. |
| Code-search corpus | 180 verbatim merged-PR queries across three public repositories pinned at fixed SHAs | Real development text avoids prompts invented around what the current search system already handles well. |
| Code-search judgments | Two independent blinded reviews, disagreement-only adjudication, and an all-query sensitivity view | The score is based on inspected pinned source, not files mechanically inferred from the PR that supplied a query. |
| Fixed-pool limit | Equal-depth top-10 pooling from every published project-scoped lane | Paths outside the frozen pool are unjudged, not proven irrelevant. A new system or changed ranking must be pooled and reviewed again. |
| Judge comparability | ContextStream's LongMemEval numbers use the official gpt-4o-2024-08-06 autoeval | Some vendors score with a different judge: Zep's 90.2% uses gpt-5.4 as reader and judge, ByteRover's 92.8% a Gemini judge, and Mem0 does not state its judge. A different judge is a different comparability class, not a head-to-head; we flag it rather than absorb it into the same table silently. |
| Wake-bench isolation | Fresh clone and fresh session per agentic task run | Keeps the project-context benchmark separate from conversational recall. |
| Limited scope | One wake-bench repo, one agent model, three trials per cell | Wake-bench single-task deltas are directional; LongMemEval is the standard full-suite number. |
Your project already produces this knowledge. Keep it.
ContextStream captures decisions, fixes, lessons, and conventions once, then gives them to the next agent at the start of work.