Memory That Outlives the Session

An agent's memory problem is not storage. The context window is finite and the store is not, so every turn you have to answer a harder question: which few things does it need now?

mnemosyne is my answer. The first constraint I set was that it must not take ownership of the data. Markdown files stay the source of truth, with plain frontmatter, readable in an editor, greppable and diffable, and still consumed by the built-in memory features I did not want to break. Everything mnemosyne adds sits on top of that: a SQLite shadow index, local embeddings, an MCP server, and lifecycle hooks.

If the index is ever corrupt or stale, you delete it and reindex, and nothing is lost. I value that property more than any retrieval improvement.

Two retrievers, fused

Recall runs a lexical query and a semantic query, then fuses them with reciprocal-rank fusion.

• FTS5 BM25 finds the memory that literally contains ANTHROPIC_VERTEX_PROJECT_ID.
• bge-small embeddings find the memory about "the GCP auth thing" when you asked about credentials.

Neither is much use without the other. Keyword search is helpless at paraphrase, and vector search is confidently wrong about identifiers and error strings, which happen to be most of what you want to look up in a codebase. RRF is the least clever way to combine them, since it reads ranks rather than scores and the two retrievers never need calibrating against each other. I have not found a reason to replace it.

The embeddings run locally, with no API key, no network, and no torch in the dependency tree. If the embedding backend is unavailable, recall degrades to BM25-only instead of failing. That degradation is the feature. An offline memory system that returns nothing is worse than a dumb one that returns something.

Similarity is brute-force cosine over the whole set. At hundreds to low thousands of memories that is nowhere near the bottleneck, and a vector index would be a dependency bought against a problem I do not have yet.

The hook is the actual product

Retrieval quality matters much less than when retrieval happens.

The load-bearing piece is a hook on prompt submission that injects the top few relevant memories for that specific prompt, rather than the whole index or a summary of it. Loading everything every turn is the obvious implementation and it is exactly wrong. It spends the budget you were trying to save, and it buries the three useful facts in three hundred.

Capture is hooked the same way. A [REMEMBER: ...] marker written mid-reply gets persisted when the turn ends, and an end-of-session sweep distils durable facts as a safety net for the ones nobody marked.

Surviving compaction

Here is the case that made me build this. A long session gets compacted, and the summary keeps the narrative while dropping the specifics: the port number, the decision that was already argued about and settled, the one gotcha that cost an hour.

So compaction is a hook too. Before it runs, mid-session facts are persisted to disk and a small set of anchors is re-emitted into the new context. The session forgets the transcript, which is the entire point of compacting, but it does not forget what it learned.