Cross-Session Memory for AI Agents: Mem0, SQLite, or a Custom ONNX Server?

Deutsche Version

Besides an MCP server for Zotero, I use a second, thematically independent MCP server for AI coding agents like Kimi Code: a cross-session memory. The two building blocks complement each other, but can also be used independently. This article documents which alternatives I evaluated, which solution I chose, and why.

Developed by Michael Logies in collaboration with Kimi Code, August 2026.

Starting point: which memory files already exist?

At every start Kimi Code reads three kinds of memory files:

These files work well for explicit, long-term rules. They are weak, however, in two respects:

1. Unstructured or semi-structured information – for example: “Three weeks ago the user mentioned they prefer X.” Such facts have to be manually copied into the right Markdown file.
2. Retrieval across many sessions – When information is scattered across MEMORY.md, it has to be read or searched at startup. As the volume grows, this becomes inefficient.

What was the problem?

Despite the memory files, cross-session memory remained limited because:

• not every piece of information is manually entered into a Markdown file,
• searching static files is not semantic,
• the context of a running session is compressed – details that are not in the memory files may be lost.

Alternatives evaluated

1. Expanding Markdown files only

The simplest option: more files, better structure, longer MEMORY.md.

• Advantage: no installation, fully transparent, version-controllable.
• Disadvantage: manual maintenance, no semantic search, poor scalability.

2. Mem0 / OpenMemory MCP server

Mem0 is a well-known memory service for AI agents.

• Problem: The official mem0-mcp is archived and cloud-centric. A local OpenMemory branch was no longer findable – only redirects in the documentation.
• Conclusion: not suitable for purely local, offline-capable operation.

3. Custom SQLite server without embeddings

A simple database with full-text search.

• Advantage: lean, offline, fast.
• Disadvantage: no semantic search; exact keywords would be required.

4. Memory-MCP server with PyTorch / Transformers

A custom Python server with fastmcp + sentence-transformers + sqlite-vec.

• Advantage: local embeddings, semantic search, fully offline.
• Disadvantage: The Python environment alone needed about 1.4 GB. After temporary files, only about 900 MB of free space remained, so the attempt was abandoned.

5. Hindsight (Vectorize)

Hindsight is probably the most powerful alternative evaluated: instead of storing only text embeddings, it extracts structured facts, resolves entities, builds a knowledge graph from them, and automatically updates “mental models”. Its three core operations are aptly named retain (store), recall (search), and reflect (infer), supplemented by cross-encoder reranking of search results.

• Advantage: much more than pure similarity search – structured knowledge representation that also maps relationships between facts.
• Disadvantage (setup effort): Hindsight is not a single script but a small service cluster (including backend service, dashboard, MCP server) that requires a PostgreSQL database with a vector extension (pgvector or similar) and is typically run via Docker Compose – installation and configuration (.env, database connection, possibly multiple containers) are considerably more complex than a single Python venv.
• Disadvantage (resource requirements): For knowledge-graph extraction, Hindsight additionally needs an LLM – either a paid cloud API per storage operation or a local model via Ollama. For local processing in convincing quality, the vendor itself names significantly larger models (up to variants requiring around 80 GB RAM or a dedicated GPU); smaller local models also run, but with reduced extraction quality. Compared with the chosen solution (a roughly 275 MB venv plus a 112.8 MB embedding model (multilingual-e5-small), with no additional LLM call per note), this is a substantially larger footprint.

For the Linux VM without GPU and limited disk space, Hindsight was therefore not an option – not because it is technically weaker, on the contrary, but because effort and resource requirements exceeded what was necessary for the intended purpose (a lean, quickly retrievable session memory).

6. Memory-MCP server with ONNX (the chosen solution)

Instead of PyTorch, the server uses onnxruntime with a quantized, multilingual embedding model.

• Venv only about 275 MB instead of 1.4 GB.
• Model 112.8 MB (multilingual-e5-small, 12 layers, multilingual).
• Runs on CPU – the Linux VM has no GPU.
• Fully offline, with hybrid search (vector + FTS5).

The chosen solution in detail

Available tools

Why ONNX instead of PyTorch?

1. Disk space: After the failed PyTorch attempt, it was clear that only a lean solution could work.
2. Offline operation: The server should not require a cloud connection.
3. Semantic search: Unlike pure Markdown files or SQLite full-text, it enables retrieving information by concept similarity rather than exact keywords.
4. CPU-only: The Linux VM has no GPU support.
5. Integration into the AI client: As an MCP server, memory is automatically available after startup.

Expectations and limits

Short term

• Default storage for information relevant across sessions.
• Fast retrieval of notes via memory_search using keywords or concepts.
• Complement to AGENTS.md / MEMORY.md / WORKLOG.md, not a replacement: permanent rules and technical details remain in the Markdown files, cross-session notes go primarily into the Memory-MCP.

Medium term

• Better continuity for recurring topics, such as Zotero workflows, prompt versions, or troubleshooting.
• Less manual copying of information into Markdown files.
• Ability to reference earlier notes for new tasks.

Limits

• The quality of semantic search depends on the small all-MiniLM-L6-v2 model. It is not as powerful as large embedding models, but sufficient for retrieving notes.
• The server stores what it is given. It does not replace active summarization or planning.
• Tags help with filtering, but require disciplined maintenance.

Changelog

• 2026-09-03 – Chunked embedding (no more 512-token cutoff): Notes longer than ~450 tokens are split at markdown boundaries into chunks of at most 450 tokens (with 50-token overlap if a single paragraph exceeds the limit) and embedded in full. Previously only the beginning of a long note was represented in the vector (max. 512 tokens) — semantic hits located deep inside a long note stayed invisible. Search aggregates chunk hits per note (minimum distance) and fuses them with full-text search as before; memory_status now also reports chunk_entries. The download package has been updated to this version.
• 2026-09-03 – Knowledge files updated: Zotero-MCP-Knowledge brought up to the Zoteus v1.12.0 state (semantic search now fully via Zoteus: 0.33–0.84 s/query instead of 93–105 s; 429 throttle dials 256/8000; new gotchas: create_items quoting and page verification), usage-rules timeline extended (2026-08-31–2026-09-03).
• 2026-08-31 – Automatic backups: setup.sh now sets up an automatic hourly backup job — snapshot of the memory database only when it changed, 8-day retention, restore with a single cp command. No action needed; requires sqlite3, which the script installs if necessary. Also new: a verification routine (verify.sh) that checks the venv, Python packages, model files, database, backup job and disk space, and writes a Markdown report to ~/.qwen/memory-install-report.md for the AI to evaluate. (This also fixes the previous model download via huggingface-cli, which silently broke with huggingface_hub ≥ 1.x and now uses the Python API.)
• 2026-08-30 – Server overhaul (now 7 tools): New tool memory_update (edit notes in place; re-embedding and index sync only when content changes). Self-healing at server start: the FTS index is rebuilt from the content table and missing vectors are re-embedded — both are now explicitly committed (the old backfill was lost when the connection closed because it was never committed). FTS sync on delete/update fixed: with external-content FTS5 the index receives the old content via the ‘delete’ command before the row disappears — the old order left orphaned index entries (empirically confirmed). memory_search quotes search terms individually and joins them with OR instead of treating the whole query as a phrase (better recall for multi-word questions); tokenizer truncation at 512 tokens (E5 limit, prevents ONNX crash on long notes); new DB path override MEMORY_DB_PATH for tests/portable use; SQLite timeout 30 s against “database is locked”. The download package has been updated to this version.
• 2026-08-28 – memory_search bug fix: Hits that were found only via full-text search (FTS5) but not via the vector search caused the entire search to abort with a NoneType error. Such hits now return vector_distance: null instead of crashing. The download package has been updated to this version (including the portable server build with current paths).
• 2026-08-25 – Hybrid search v2: Switched to multilingual-e5-small, full-text search via FTS5, RRF fusion, temporal decay disabled.

Download: Memory-MCP Server as a Setup Package

The complete setup (server script, installation script, configuration template, documentation) is available as a ZIP package for self-installation on your own Linux system:

📦 memory-mcp-setup.zip (24 KB, as of 2026-09-05)

New: The package now sets up automatic hourly backups of the memory database (only when changed, 8-day retention) — your assistant backs up its own memory. My assistant also keeps its environment up to date fully automatically: it updates the semantic Zotero index daily (Zoteus, text-embedding-3-small, incremental with full text) and runs a weekly update check for Zoteus, Zotero, and the plugins — see the automatisms paragraphs on the AI Workstation page (§6.3, §12). As of 2026-09-04, the package also contains the usage notes for beginners: the setup script copies usage-notes.md (or nutzungshinweise.md) to ~/.qwen/, and the §3.2 assignment in the AI Workstation guide has your agent save it as a memory note and show it at every start.

📄 Zotero-MCP Knowledge for AI Agents (39 KB, as of 2026-09-03) – Condensed hands-on experience, workflow rules, tool comparisons, and benchmark results. A Markdown file that any AI agent can import directly into its memory system.

📜 Memory-MCP Knowledge: Usage Rules & Learning Protocol (11 KB) – All rules for operating the memory MCP server: the memory-first mandate, storage and care rules, tag discipline, and the learning timeline with all dates. A Markdown file ready to import into any memory system.

Conclusion

The Markdown files remain the foundation for rules and technical documentation. The Memory-MCP server adds a lean, offline-capable, semantic note database. The decision for ONNX instead of PyTorch was crucial to reduce memory requirements to an acceptable level.

If you want to combine this approach with an MCP server for Zotero, the technical setup is described on the page Zotero MCP Server vs. Beaver. Experiences with Kimi Code as the AI client can be found under From Claude Code Sonnet to Kimi Code K2.7-code. For a quick start with Qwen Code including Memory-MCP setup (one to two hours), see the page An AI Agent in One to Two Hours.