Benchmark: LongMemEval (ICLR 2025)
Evaluated on LongMemEval, the industry standard benchmark for long-term conversational memory. The benchmark contains 500 human-curated questions across 5 core capabilities, testing whether an assistant can retrieve and use long interaction histories instead of relying on the current chat only.
Benchmark Setup
| Field | Value |
|---|---|
| Benchmark | LongMemEval (ICLR 2025) |
| Dataset | 500 human-curated questions |
| Variant | LongMemEval_S, about 115k tokens per question |
| Method | Hybrid RRF, BM25 plus Semantic Vector Search |
| Embedding | multilingual-e5-small (local daemon's production model) |
| LLM calls | 0, pure retrieval with no generation cost |
| Pipeline | Real daemon retrieval (unifiedCascadeSearch + live SQLite + FTS5 + embeddings) |
| Runtime | 35 minutes total (500 questions, 4.2s/question) |
Updated 2026-08-06: results below are measured through the real local-daemon retrieval pipeline (SearchEngine.unifiedCascadeSearch), not an independent Python harness. Each question builds an isolated index, embeds its haystack with the same E5 model the daemon uses, and runs the exact recall code path production serves. This is the closest measurement to what users actually get.
Awareness Memory Results
| Metric | Score | Hits | Note |
|---|---|---|---|
| Recall@1 | 80.2% | 401 / 500 | |
| Recall@3 | 92.8% | 464 / 500 | |
| Recall@5 | 96.0% | 480 / 500 | Primary metric |
| Recall@10 | 98.6% | 493 / 500 |
R@5 Leaderboard
| System | R@5 | Note |
|---|---|---|
| MemPalace (ChromaDB raw) | 96.6% | R@5 only |
| Awareness Memory (Hybrid, daemon path) | 96.0% | Hybrid RRF, real production pipeline |
| Awareness Memory (earlier harness) | 95.6% | Independent Python harness (2026-04) |
| OMEGA | 95.4% | QA Accuracy |
| Mastra (GPT-5-mini) | 94.9% | QA Accuracy |
| Mastra (GPT-4o) | 84.2% | QA Accuracy |
| Supermemory | 81.6% | QA Accuracy |
| Zep / Graphiti | 71.2% | QA Accuracy |
| GPT-4o (full context) | 60.6% | QA Accuracy |
MemPalace 96.6% is Recall@5 only, not QA Accuracy. Palace hierarchy was not used in that evaluation. The daemon-path number (96.0%) is measured on the actual production retrieval code, which makes it the most representative number.
R@5 by Question Type
| Question type | R@5 |
|---|---|
| multi-session | 99.2% |
| knowledge-update | 98.7% |
| single-session-asst | 98.2% |
| temporal-reasoning | 93.2% |
| single-session-user | 92.9% |
| single-session-pref | 90.0% |
| Overall | 96.0% |
Ablation Study
| Retrieval method | R@5 | What it shows |
|---|---|---|
| Vector-only | 92.6% | Semantic retrieval alone is strong but misses exact lexical cues. |
| BM25-only | 91.4% | Full-text retrieval is strong but misses paraphrases. |
| Hybrid RRF | 95.6% | Hybrid improves by about 3 points over either single method alone. |
Ablation numbers are from the 2026-04 harness run; the daemon-path run (96.0%) includes additional exact-title-phrase handling that lifts Recall@1 further.
Method Notes
This benchmark was run with the open-source local version, Awareness-Local. Awareness Memory uses Hybrid RRF retrieval: BM25 full-text search plus semantic vector search, fused without LLM calls on the retrieval path. The production daemon uses multilingual-e5-small embeddings (multilingual 384-dimension class), and the LongMemEval_S variant with about 115k tokens per question.
The lightweight embedding model was chosen deliberately so ordinary laptops can run the product locally. In practice, a stronger embedding model and the full cloud version should have a higher ceiling than this local lightweight setup.
Recall@5 is a retrieval metric: it means the relevant evidence appears in the top five retrieved memories. It is not the same thing as final QA accuracy. The leaderboard above keeps that distinction explicit because some public numbers are retrieval-only while others are QA accuracy.
Reproducibility: the daemon-path runner is benchmarks/longmemeval/run_f053_daemon_path.mjs — it drives the real unifiedCascadeSearch per question with an isolated index, so the number is reproducible on any machine with the SDK checked out.
References:
- LongMemEval paper: https://arxiv.org/abs/2410.10813
- LongMemEval repository: https://github.com/xiaowu0162/LongMemEval
- Awareness-Local: https://github.com/everest-an/Awareness-Market
- Awareness benchmark runner: https://github.com/everest-an/Awareness/tree/main/benchmarks/longmemeval