01 — Our run
Awareness on LongMemEval_S (500 questions)
Environment: Apple M1 / 8GB RAM, all-MiniLM-L6-v2 embedding (23MB ONNX), hybrid RRF (vector 0.7 / BM25 0.3, k=60), session-level indexing. Full run: 843 seconds. Dataset: LongMemEval_S, ~115k tokens and ~47 sessions per question.
Recall@1
77.6%
388 / 500
Recall@3
91.8%
459 / 500
Recall@5
95.6%
478 / 500
Recall@10
97.4%
487 / 500
Ablation — hybrid beats either signal alone
| Method | Recall@5 |
|---|---|
| Vector-only | 92.6% |
| BM25-only | 91.4% |
| Hybrid RRF (0.7 / 0.3) | 95.6% |
By question category (Recall@5)
| Category | R@5 | Hits | |
|---|---|---|---|
| Knowledge update | 100.0% | 78/78 | conflict detection is a first-class feature |
| Multi-session | 98.5% | 131/133 | cross-session reasoning |
| Single-session (assistant) | 98.2% | 55/56 | |
| Temporal reasoning | 94.7% | 126/133 | |
| Single-session (user) | 88.6% | 62/70 | |
| Single-session (preference) | 86.7% | 26/30 |
02 — The comparison
Two metrics. We refuse to mix them.
The field quietly conflates two different things: recall@5 (did retrieval surface the right evidence?) and QA accuracy (did the final answer match the judge?). They are not comparable. We publish both, labeled.
Recall@5 — like for like
| System | R@5 | Source |
|---|---|---|
| Memax | 96.9% | docs.memax.app/quickstart/benchmarks |
| MemPalace (raw ChromaDB) | 96.6% | github.com/MemPalace/mempalace benchmarks |
| Awareness | 95.6% | this page, scripts public |
| Stella V5 1.5B (dense) | 70.0% | LongMemEval paper |
| BM25 (sparse) | 51.0% | LongMemEval paper |
We are 1.3 points behind Memax and 1.0 behind MemPalace on recall. We run locally on an M1 with no LLM calls at retrieval; they require a hosted stack. Trade-offs are the honest unit of comparison.
QA accuracy — labeled as a different metric
| System | QA | Generator | $/Q |
|---|---|---|---|
| agentmemory | 96.2% | Claude Opus | ~$0.170/Q |
| PwC Chronos | 95.6% | enhanced config | $0.10–0.54/Q |
| Mastra OM | 94.9% | GPT-5-mini | ~$0.130/Q |
| Backboard | 93.4% | GPT-4.1 | ~$0.070/Q |
| Ensue | 93.2% | GPT-5-mini | ~$0.003/Q |
| OMEGA | 93.2% | GPT-4.1 | ~$0.022/Q |
| MemMachine | 93.0% | GPT-5-mini | ~$0.005/Q |
| Hindsight | 91.4% | Gemini-3-Pro | ~$0.650/Q |
| Memax | 91.2% | Claude Sonnet | ~$0.046/Q |
| Supermemory | 85.2% | Gemini-3-Pro | ~$0.110/Q |
| Zep / Graphiti | 71.2% | GPT-4o | ~$0.058/Q |
| Full-context baseline | 60.2% | GPT-4o | ~$0.290/Q |
| Mem0 (independent eval) | 49.0% | — | — |
Source: aggregated self-reports compiled by docs.mem0-independent comparisons (see footnote). QA accuracy is dominated by the generator model — it measures the answer, not the memory. Awareness does not yet publish a QA number; we will, with the generator and judge pinned.
What the incumbents self-report
LoCoMo
92.5
LongMemEval (QA)
94.4
BEAM 1M
64.1
BEAM 10M
48.6
Mem0's self-reported numbers (mem0.ai blog, April 2026). An independent LongMemEval evaluation of Mem0 measured 49.0% QA accuracy. Self-reported numbers across the field should be read with their source attached — which is what this page does.
03 — Next frontier
LongMemEval-V2 has an official leaderboard. We are submitting.
Released May 2026, LME-V2 raises the bar: 451 curated questions, up to 115M-token histories, and a latency-adjusted frontier score (LAFS) instead of a single number. Published baselines show how much headroom remains:
| Method | Accuracy (Small) |
|---|---|
| No retrieval | 1.3% |
| RAG: query → slice | 42.8% |
| AgentRunbook-R | 58.6% |
| Codex (off-the-shelf) | 69.9% |
| AgentRunbook-C | 74.9% |
Our LongMemEval-V2 submission is in preparation. When it lands, the result — whatever it is — will be added to this page with the same disclosure standard.
04 — Full disclosure
Methodology, sources, and limits
Reproduce it yourself
- Scripts:
benchmarks/longmemeval/run_benchmark.pyin the open repo - Dataset:
xiaowu0162/longmemeval-cleanedon HuggingFace - Hardware for this run: Apple M1, 8GB RAM — no GPU, no hosted API
- Embedding: all-MiniLM-L6-v2, 384-d, quantized ONNX
Known limits
- Single benchmark, single dataset size (LongMemEval_S, not _M).
- Session-level granularity; turn-level indexing is a known improvement path.
- Retrieval only — end-to-end QA accuracy not yet published by us.
- One embedding model tested; stronger models may shift all rows upward.
Why we publish numbers we do not win: memory benchmarks are the only way developers can trust what they are installing. If every vendor only shows the table they top, the field learns nothing. Our advantage is not one point of recall — it is that you can verify everything on this page from an M1 laptop, and that memory itself is free, local-first, and verifiable on-chain. Judge us on that.
Give your AI a memory that actually persists.
Free forever. Local-first. One command: npx @awareness.market/setup