# Awareness Benchmark System Card v0.1

**System:** Awareness local memory daemon
**Benchmark:** LongMemEval-S
**Date:** 2026-10-29
**Canonical:** https://awareness.market/en/benchmarks
**Protocol:** https://awareness.market/protocol/awareness-memory-protocol-v0.1.md
**Reference implementation:** https://github.com/everest-an/Awareness-SDK
**Contact:** e@awareness.market

---

## 1. What this card covers

This card documents the retrieval benchmark we publish for Awareness: LongMemEval-S, 500 questions, Recall@5, measured through the real local-daemon path rather than a separate synthetic harness.

It covers:

- the system and hardware used;
- the retrieval pipeline;
- the metric and its limits;
- the result we report;
- the known limitations and failure cases;
- how to reproduce it;
- how to challenge it.

It does **not** claim that the benchmark proves end-to-end answer quality. Recall@5 is a retrieval metric.

## 2. System under test

- Local-first memory daemon with a SQLite index and Markdown storage.
- Hybrid retrieval: BM25 full-text search + vector embeddings, fused with reciprocal rank fusion (RRF).
- Retrieval runs with **zero LLM calls**.
- The benchmark exercises the same daemon pipeline used by the product, not a mock or a separate in-memory index.

## 3. Benchmark

**LongMemEval-S**, 500 questions, approximately 115k tokens of context per question.

- **Metric:** Recall@5. A question is counted as a hit if at least one required memory is present in the top five retrieved results.
- **Reported result:** **96.0% Recall@5**.
- **Hardware:** Apple M1, 8 GB RAM.
- **Latency:** approximately 4.2 seconds per question on the reference machine.
- **Retrieval cost:** zero LLM calls; embeddings and index are local.

## 4. Methodology

1. Load the LongMemEval-S questions and their memory sessions.
2. Index the memory sessions with the same retrieval stack used by the local daemon.
3. Run the question through the real retrieval path (not a bespoke benchmark retriever).
4. Take the top five results.
5. Count a hit if at least one required session appears in the top five.
6. Aggregate Recall@5 across the 500 questions.

The benchmark script is in the repository:

- `benchmarks/longmemeval/run_f053_daemon_path.mjs`
- `benchmarks/longmemeval/` (supporting data and scripts)

## 5. Result

| System | Benchmark | Metric | Result | Local / zero-API | Notes |
| --- | --- | --- | ---: | --- | --- |
| **Awareness (daemon path)** | LongMemEval-S | Recall@5 | **96.0%** | Yes | Real daemon pipeline, M1/8GB, ~4.2s/query |
| Mem0 | LongMemEval | Reported R@5 / QA accuracy | 49.0% (OSS) / 93.4% (self-reported) | Graph on paid tiers | Public numbers use different setups |
| Zep | LongMemEval | Reported | 63.8% | SaaS | Temporal graph, cloud-focused |
| MemPalace | LongMemEval | R@5 | 96.6% | Yes | R@5 only, raw verbatim |

Comparisons are directional. Do not compare a Recall@5 number with a QA-accuracy number and call it a win.

## 6. Limitations

- **Retrieval is not answer quality.** Recall@5 measures whether the right memory is retrieved, not whether the final answer is correct.
- **One benchmark is not the product.** LongMemEval-S does not cover every memory type, language, or workflow. It is one signal.
- **Self-reported.** This result is produced by the Awareness team. It has not yet been independently replicated by a third party. Treat it as a reproducible claim, not an external validation.
- **Dataset and contamination.** We have not audited LongMemEval for training-set contamination. The result should be read as a benchmark result under the published methodology, not as a universal capability claim.
- **Hardware dependence.** Latency numbers are from the reference M1/8GB machine. Retrieval quality may change with a different embedding model, index, or configuration.
- **No failure-case table yet.** v0.1 documents the known limitations but does not publish a per-question failure analysis. A v0.2 should add the failure breakdown.
- **No independent reproduction yet.** We invite one and will link it here when it exists.

## 7. Reproduction

From the repository:

```bash
git clone https://github.com/everest-an/Awareness-SDK
cd Awareness-SDK
node benchmarks/longmemeval/run_f053_daemon_path.mjs
```

Requirements and dataset details are documented in `benchmarks/longmemeval/`. Reproduce on your own hardware; do not assume the same latency.

If you reproduce it, send the command, hardware, commit hash, and result to e@awareness.market. We will link independent results here, including results that disagree.

## 8. What would change our claim

- An independent reproduction that does not reach 96.0% under the documented methodology.
- A finding that the benchmark data is contaminated or misconfigured.
- A retrieval configuration change that moves the result materially.
- A better benchmark for agent memory that we should adopt.

## 9. Citation

```
Awareness Benchmark System Card v0.1, 2026-10-29.
https://awareness.market/benchmarks/awareness-system-card-v0.1.md
```

Do not cite this as "the best memory system." Cite it as "Awareness reports 96.0% Recall@5 on LongMemEval-S through its local daemon path; independent replication is pending."
