The Memory Server in the Lab: 21 Studies, 125 Test Questions, +20.8 Percentage Points
As of 2026-09-20 · Scope: 125 recorded real search queries in three measurement baskets · 21 research papers evaluated in full text · all numbers documented in the reports linked below
The Memory-MCP main page describes how the memory server is built. This page documents how we measured and improved its retrieval quality — guided by the research literature and proven by our own measurement series.
The starting question
The server answers queries by mixing two paths: a full-text search (finds exact terms) and a semantic search (finds meaning, even when the wording differs). Both result lists are fused into one.
The question was: How reliably does the server retrieve an old note when the user phrases the query in different words than the note used? — And: Which levers improve that measurably?
How we measured
- 125 recorded real-world queries from the agent's daily work, in three "baskets": 46 questions whose answer notes were involved in the tuning (basket 1, "sharpened yardstick"), 42 fully independent questions (round 2), and 37 questions covering the hard remainder (round 3, a deliberately tough vocabulary field).
- Gold answers: for every query, a database check verified which note(s) contain the correct answer. We measured whether it lands in the top 5 hits (Recall@5) and how high up (MRR).
- Every lever was measured in isolation — before/after with an identical query set, so the effect can be attributed to the lever, not to chance.
The 21 studies (plus one foundational paper)
All papers were identified via OpenAlex and — except one — evaluated in full text. The right column summarizes what our system made of them.
| Paper (year) | What our system took from it |
|---|---|
| Reciprocal Rank Fusion, Cormack et al. (2009) | The basis of our fusion; the grid experiment showed K is uncritical — the weight ratio of the two search paths is the real lever |
| MemGPT, Packer et al. (2023) | Context managed like OS memory; external note stores are the right way to survive across sessions |
| Generative Agents, Park et al. (2023) | Importance thresholds as a consolidation trigger — a CPU-friendly pattern for the cleanup cadence |
| RECOMP, Xu et al. (2023) | Compressing retrieved content saves context but loses precision — we return full notes plus a text snippet as preview |
| E5, Wang et al. (2022) | The contrastively trained model family behind our multilingual embedding model |
| Lost in the Middle, Liu et al. (2024) | Hit position in context matters: far-down results degrade — so few, well-ranked hits beat many |
| Dense X Retrieval, Chen et al. (2024) | Fine-grained retrieval units (sentences/facts) — the template for our key lines as a fact layer |
| The Chronicles of RAG, Finardi et al. (2024) | Chunking is among the biggest levers — it motivated our measurement series on splitting behavior |
| Late Chunking, Günther et al. (2024) | Contextual chunk embeddings need long contexts; not feasible within our 450-token limit |
| Meta-Chunking, Zhao et al. (2024) | LLM-based segmentation is strong but unaffordable on CPU (140 to over 57,000 seconds per document in the studies) |
| Mix-of-Granularity, Zhong et al. (2024) | Two granularities beat one; the gain appears when moving from the first to the second level |
| LongMemEval, Wu et al. (2024) | A benchmark with a clear message: title-like key lines yield +9.4 % recall — the template for our search anchor |
| LoCoMo, Maharana et al. (2024) | Very long conversation archives; replicates the ground rule: never truncate the note body |
| MemoryBank, Zhong et al. (2024) | Ebbinghaus-style forgetting; we deliberately disabled time decay because our knowledge is time-stable |
| A Survey on Memory Mechanisms, Zhang et al. (2024) | Taxonomy of memory mechanisms — helped us choose levers systematically instead of guessing |
| RAPTOR, Sarthi et al. (2024) | Hierarchical summary trees; rejected in review: the risk of summary layers inventing facts does not fit a fact store |
| HippoRAG, Gutiérrez et al. (2024) | Graph-based stores with neuroscience inspiration; deliberately not built without a GPU budget |
| Zep, Rasmussen et al. (2025) | Two timelines (valid-from vs. known-at); our simple timestamps plus disabled decay cover our needs |
| A-Mem, Xu et. al (2025) | Agentic note linking; confirmed that robust smaller embedding models suffice for note processing |
| Mem0, Chhikara et al. (2025) | The ADD/UPDATE/DELETE/NOOP decision at write time: "replace only if richer" — the pattern behind our supersede rule |
| From RAG to Memory, Gutiérrez et al. (2025) | Non-parametric learning via note stores — the architecture class our system implements in practice |
| Memory Construction, Pan et al. (2025) | Segment-level storage; key measurement: merging at write time clearly beats merging at query time |
The measurement series
Retrieval rate = Recall@5 (is the right note among the top 5 hits?). MRR additionally rewards how high up it lands.
| State (2026) | Basket 1 (46 queries, tuned) | Round 2 (42, independent) | Round 3 (hard field) |
|---|---|---|---|
| Baseline (Sep 19) | 89.1 % / MRR 0.696 | — | — |
| After cleanup + search anchors (Sep 19) | 97.8 % / 0.864 | 83.3 % / 0.707 | 68.4 % / 0.564 |
| Search weighting 0.85 + further cleanup (Sep 20) | 91.3 % / 0.815 * | 88.1 % / 0.766 | 73.0 % / 0.587 |
| + Translation dictionary (measured in isolation) | 91.3 % / 0.815 | 88.1 % / 0.766 | 75.7 % / 0.614 |
| + Key-line enrichment = final state | 91.3 % / 0.817 | 88.1 % / 0.766 | 89.2 % / 0.731 |
* Basket 1 only "dropped" because the yardstick was sharpened: accidental hits on meanwhile-deleted duplicate notes no longer count as success, and one question deliberately targets a removed old state. The independent baskets show the real gain.
Overall result for the hardest basket: from 68.4 % to 89.2 % retrieval rate (+20.8 percentage points), MRR from 0.564 to 0.731. A control run without the new dictionary reproduced the old values exactly — the gain is attributable to the lever.
The implemented improvements
- Clean up instead of hoard: 446 → 403 notes (outdated ones deleted, full texts archived in files); 28 oversized notes reduced to a short version plus file pointer. Measurement confirmed: this improves ranking.
- Key lines as search anchors: The first line of every note is weighted 3× in search and additionally fed into the embedding model. The largest single gain of the program.
- Search weighting: The keyword arm counts 0.85× instead of equally — this fixes fusion losses on questions only one search path can answer.
- Translation dictionary: A maintained dictionary (
synonyms.json) invisibly translates everyday words into domain terms ("vault" → password database, "thinking level" → reasoning). Further bridges are added gradually — checked by a health-check script that regularly replays the measurement baskets. - Comfort: Every hit now shows its best text snippet, and the search can operate with a token budget.
- Deliberately rejected: a date field in search (cost precision, gained nothing), changes to the fusion constant (no effect), hierarchical summary trees (hallucination risk in a fact store).
Downloads
📄 Research report (20 KB, as of 2026-09-19) – The full project report: research question, methodology, all measurements, rejected alternatives.
📄 Full-text analysis (18 KB, as of 2026-09-19) – The consolidated evaluation of all 21 full texts with source references.
The setup guide for the memory server and the usage rules are on the Memory-MCP main page (German version available there as well).