I Let My Coding Agent Build MemPalace on Its Own, but Had to Use Classical Engineering to Get It Really Working

🎥 Watch the video on YouTube

Why I wanted my own memory store

If you use coding agents every day, you probably end up trying to give them some kind of persistent memory. With Hermes Agent, you can watch it evolve skills on the fly during ordinary conversations to handle recurring workflows better. That was great to see, but Hermes was not my only tool. I regularly switch between Hermes, Antigravity, Reasonix, and WorkBuddy depending on what I am doing.

That immediately ran into a wall: Hermes tucks the skills it learns into its own internal storage instead of the standard $HOME/.agents/skills directory. Even after setting up a rule to mirror those skills into my workspace root, the memory remained stuck in Hermes's own harness. None of my other agents could see it.

Around the time Milla Jovovich (the actress who played Alice in the Resident Evil movies) released a project called "MemPalace", the name stuck with me. I decided to build my own version for my workstation. I did not look at her repository or borrow any code; the name was just a fun coincidence, and what I needed felt simple enough that I did not want outside architecture getting in the way.

I also had a second, more cynical motive: I wanted to see what would happen if I actually took the "vibe coding" hype seriously. Everyone was claiming software architecture was obsolete and you could just talk an LLM into building working systems without classical engineering discipline. I wanted to see where that broke down.

My workspace on Windows is an active environment. Over months of work, it piles up helper scripts, refactors, architecture decisions (ADRs), and specific workarounds. People sometimes describe this as "raising a lobster" in OpenClaw or babysitting a long running Hermes profile. But when context windows fill up and model endpoints forget everything between requests, an agent without memory keeps making the exact same mistakes. It stumbles on the same Windows file locks, writes bad systemd timers on remote Linux boxes, and misconfigures the same background daemons.

I needed four things:

1. Shared across agents: Hermes, Antigravity, Reasonix, and WorkBuddy had to query the exact same local store.
2. Plain text ownership: Plain Markdown files on disk had to be the source of truth, not a hidden database file or a cloud service.
3. Sub-second CLI calls: A pre-task memory search had to return in milliseconds, not pause the session for ten seconds.
4. Local Windows execution: Zero cloud dependencies, zero external vector SaaS subscriptions, and no heavy background daemons.

The naive experiment: letting the agent build it

I gave my coding agent a completely open prompt: write a persistent memory and search tool for this workspace so any agent can pull up past solutions before starting work. I gave it no schema, no test cases, no retrieval benchmark, and no quality gate. I let it pick its own path.

For storage, I told it to skip vector databases and use DuckDB. The viral agent-memory projects at the time were dragging in ChromaDB, complex hierarchical "wings" and "palaces", and dozens of dependencies. For a single developer workstation, running an external vector database sounded ridiculous. DuckDB seemed cleaner: Suppose I had an MCP server for it, a quick uv add duckdb gave me in-process SQL without background daemons, and I was curious to test it. The plan was simple: I would write notes in Markdown, and the agent would sync them into MemPalace/memory.duckdb.

The agent agreed immediately. It told me DuckDB was lightweight, fast, and a great fit, and within minutes it wrote two standalone Python scripts: scripts/sync_memory.py to copy Markdown files into DuckDB, and scripts/query_memory.py to search them.

The scripts ran on the first try. Zero syntax errors, clean terminal output, no warnings. In less than an afternoon, I had what looked like a working memory store without designing a database or writing a line of linear algebra.

Then I tried relying on it in real work.

The autopsy of V1: digital theater and 0.06 recall

For weeks, every agent I ran dutifully called query_memory.py before starting a task. Seeing the query fly by in the terminal gave me the comfortable feeling that the agents were learning.

In practice, the search was almost useless. The CLI either returned nothing at all or dumped twenty irrelevant notes onto the screen. My agents kept walking into the same Windows file locking bugs I had spent hours documenting. The memory query had become pure theater: burning tokens and machine time while giving the agent zero actionable context.

During one debugging session where the agent did something stupid that was already documented in my notes, I stopped and asked it directly:

"Can MemPalace do semantic search right now?"

The agent replied with one word:

"No."

I stepped out of the repository entirely. I opened a fresh session in web Gemini with zero workspace context and started asking basic questions about storage engines and search.

The first issue was obvious in hindsight: DuckDB is an OLAP engine. It is built for scanning columns across millions of rows to compute analytical aggregations. An agent looking up a past tip before a task is an OLTP workload: sub-millisecond point lookups and single-row updates across maybe 150 documents. Running DuckDB for a 150-record CLI index was like hiring a cargo freighter to cross the street. SQLite had been the correct tool all along.

The second issue was my own gap in retrieval basics. Dense vectors, embeddings, and cosine distance felt like machine learning trivia I did not have time to study. But while reading engineering posts, I came across Uncle Bob Martin writing about Behavior-Driven Development (BDD). That led me into Gherkin specifications and executable contracts.

The connection was immediate: you do not need a machine learning degree to build reliable search; you need classical software engineering discipline. I did not need to invent new vector mathematics. I just needed executable acceptance tests that boxed the agent in so it could not cut corners.

When I ran an autopsy on the V1 code, the shortcuts the agent had taken were embarrassing:

1. Unranked substring scans: When asked to "add search", the agent wrote a bare SQL ILIKE '%<query>%'. No BM25 ranking, no word boundaries, no stemming. Searching for "file lock" missed notes containing "file-locking" because of the hyphen. Searching for "vmmem wslconfig" returned zero hits even though the fix was on disk. Searching for common words like "windows" dumped dozens of notes in arbitrary disk order, hiding the one note that mattered.
2. Dual-write drift: The Markdown files on disk and the rows in DuckDB were completely separate. The database only updated if I remembered to run sync_memory.py. Without automated drift checks, the database silently fell behind.
3. The 0.06 effective recall: When I finally measured V1 against twenty real developer queries, the DuckDB store only held 20% of my actual notes (27 rows in the database versus 34+ files on disk). And for the documents that were in the database, the dumb substring match found the right document only 30% of the time.

$\text{Effective Recall@5} \approx \text{Store Visibility (0.200)} \times \text{Algorithm Recall (0.300)} \approx \mathbf{0.06}$

MemPalace V1 missed 94% of the knowledge in my workspace. That was the reality of unconstrained vibe coding: code that ran cleanly, looked great in terminal demos, and failed almost completely when asked to do real work.

Rebuilding V2 with classical engineering

Fixing this did not require cloud memory frameworks like Mem0, Zep, or Letta. Those platforms add hosted databases, external API dependencies, and slow, non-deterministic LLM writes. For a local CLI workflow on Windows, they just swap one set of problems for another.

Instead, I changed how I worked with the coding agent. I stopped treating it like an architect and started treating it like a junior developer working under strict test gates.

1. The benchmark came first

Before writing any search code, I had the agent build an evaluation suite: the Golden Recall Benchmark (tests/benchmark/). We wrote 20 realistic developer queries based on actual debugging sessions in the workspace, each paired with the exact Markdown files it should retrieve. I set a hard rule: macro recall@5 had to hit at least 0.85, with zero queries returning empty. Search quality was no longer an argument or an impression; it was a pass/fail number.

2. Gherkin contracts and two test loops

To prevent the agent from improvising shortcuts, we wrote executable specifications in Gherkin (tests/features/mempalace.feature) using pytest-bdd. The agent had to implement explicit Given, When, Then scenarios covering database creation, ingestion, BM25 matching, vector cosine distance, and drift checks.

We split testing into two loops:

• Outer loop: CLI subprocess tests that ran python -m mempalace inside isolated temporary directories, checking return codes, flags, and JSON formatting.
• Inner loop: Fast in-memory SQLite tests checking FTS5 queries, vector math, and fusion ranking without touching disk.

3. Files as the only source of truth

We fixed the dual-write bug at the architectural level. The Markdown files on disk are the only real record. The SQLite database (MemPalace/memory.sqlite) is an expendable index. If you delete the database, running mempalace ingest rebuilds it from scratch. A companion command, mempalace reconcile, compares disk against the database and warns you immediately if any file is missing from the index.

4. Simple, proven components

Instead of custom search logic, we wired together four standard pieces:

• Storage: Standard library SQLite using WAL mode and busy_timeout to handle fast reads and writes cleanly.
• Lexical search: SQLite FTS5 with BM25 ranking and the Porter stemmer. Queries run with strict AND semantics by default, falling back to ranked OR only if AND returns zero hits.
• Dense vector search: Local embeddings using fastembed with BAAI/bge-small-en-v1.5 (384 dimensions) running on ONNX Runtime. No PyTorch, no CUDA setup, no GPU required. Similarities run via a vectorized NumPy cosine function.
• Hybrid fusion: Reciprocal Rank Fusion (RRF, with $k=60$) merges the BM25 ranks and vector ranks without needing score normalization:

$\text{RRF\_Score}(d) = \sum_{m \in \{\text{dense}, \text{sparse}\}} \frac{1}{60 + \text{rank}_m(d)}$

On the 20-query benchmark, BM25 alone scored 0.925 recall@5. Once combined with dense vectors via RRF, the hybrid pipeline hit 1.000 recall@5, finding the expected notes for all 20 test queries.

The 120-second freeze and operational hardening

Even with clean architecture and passing unit tests, running the system in daily work exposed an immediate failure. Right after running ingest to index new notes, running a search caused the CLI to freeze for more than two minutes.

Tracing the execution showed three compounding bugs:

1. Accidental vector wipes: ingest cleared rows in search_docs before repopulating them, and an ON DELETE CASCADE rule silently deleted all pre-computed vectors in doc_vectors.
2. Search path re-embedding: When the user ran a search, the code noticed missing vectors and tried to re-embed all 146 documents on the fly inside the query call.
3. Network calls during model load: FastEmbed and Hugging Face Hub tried to phone home to check model metadata. On a slow or restricted internet connection, that check stalled the search process indefinitely.

We fixed all three with four explicit rules:

• Stable document keys: Documents are upserted using (source_kind, source_ref). Unchanged notes keep their IDs, preserving their cached vectors across regular ingest runs.
• Fail-fast search: The search command never generates embeddings for the corpus. If vectors are missing, it falls back to BM25, returns results immediately, and warns the user to run mempalace embed separately.
• Strict offline mode: If local model weights exist, the engine sets HF_HUB_OFFLINE=1 to cut off network probes entirely.
• English-only corpus: Three legacy Chinese notes in the workspace were translated to English. Keeping the entire corpus in English eliminated cross-lingual vector distortion on the bge-small-en-v1.5 model.

Search latency after an ingest dropped from over 120 seconds down to ~2.3 seconds. Our test suite grew to 273 passing tests across BDD scenarios, unit tests, and CLI checks, running clean under ruff and pyright strict mode.

What vibe coding is actually good for

Vibe coding has a place, but a small one: scaffolding, small disposable scripts, and autocomplete for boilerplate. When you leave an LLM unconstrained, it will always pick the path of least resistance to make a prompt look solved. It will happily give you an unranked substring match on a stale database and call it "semantic search", because the script exits with code 0 and the terminal output looks convincing.

Classical software engineering is not the enemy of coding agents. It is the harness that makes them usable.

Once I stopped treating the agent like an architect and started treating it like an implementer boxed in by strict tests, the dynamic changed completely. Given Gherkin contracts, a real recall benchmark, typed data structures, and a hard boundary between indexing and querying, it built clean, working code faster than I could have typed it.

If you are building memory for your own coding agents, keep a few things in mind:

• Define your evaluation benchmark before writing search logic.
• Keep Markdown files as the source of truth, and treat the database as a throwaway index.
• Use SQLite, FTS5, and local ONNX embeddings instead of pulling in heavy vector databases.
• Keep the query path read-only and offline. No network requests, no on-the-fly embedding.
• Write executable BDD contracts so the agent cannot bluff its way around edge cases.

The code

Once V2 was running reliably in my daily work, I extracted it from my workspace's ad-hoc scripts and cleaned it up into a standalone Python package. The whole thing is open source on GitHub:

github.com/aruruka/mempalace

If you are dealing with similar cross-agent memory problems, or just want to see how the Gherkin contracts and recall benchmarks look in a real codebase, take a look. It runs completely offline on standard Python, SQLite, and fastembed, with all 273 tests and the 20-query golden benchmark included.