Parallel mining corrupts ChromaDB HNSW index — no warning, silent failure

[fubak]

Bug

Running mempalace mine on two projects simultaneously against the same palace corrupts the ChromaDB HNSW index. Mining appears to succeed (no errors during the mine), but subsequent searches and even new writes fail with an unrecoverable error.

Steps to Reproduce

# Mine two projects in parallel against the same palace
mempalace mine /path/to/project1 --wing project1 &
mempalace mine /path/to/project2 --wing project2 &
wait

# Both appear to complete without error, then:
mempalace search "anything" --wing project1
# → Search error: Error loading hnsw index

# Even re-mining fails:
mempalace mine /path/to/project1 --wing project1
# → chromadb.errors.InternalError: Error in compaction:
#   Error constructing hnsw segment reader:
#   Error creating hnsw segment reader: Error loading hnsw index

Impact

• Index corruption is unrecoverable — the only fix is to delete the entire palace and re-mine everything from scratch
• Mining itself reports no errors, giving false confidence that it succeeded
• All previously working wings in the palace are unaffected for reads, but new writes also fail — making the palace permanently read-only until wiped

Expected

Either:

1. Lock the palace during mining to prevent concurrent writes, or
2. Detect concurrent access and exit with a clear error

At minimum, document that parallel mining is unsupported.

Full Traceback

File ".../mempalace/miner.py", line 207, in add_drawer
    collection.add(...)
chromadb.errors.InternalError: Error in compaction: Error constructing hnsw segment reader: Error creating hnsw segment reader: Error loading hnsw index

Environment

• mempalace 3.0.0
• Python 3.12.3
• ChromaDB 1.5.6

[nixlim]

you can do separate palaces, that would create separate DBs and then you can mine in parallel. Obviously does not necessarily achieve the outcome that you want. But would work if you wanted to mine different projects at the same time.

I put the details on parallel mining here:
#183

still waiting for approval but can be useful

[JSee98]

@fubak
will need to test but I believe we can easily switch to the chromaDB's http client 🤔

[bensig]

Note: #544 (merged 2026-04-10) partially mitigates this by switching convo_miner.py from add() to upsert(), preventing duplicate HNSW entries on repeated mines. However, true parallel mining (two mempalace mine processes writing to the same ChromaDB collection simultaneously) remains unsafe — ChromaDB's PersistentClient is not designed for multi-process writes. Mines should be run sequentially until a storage layer with proper concurrency support is in place (#413 backend seam is now merged).

[jphein]

Updating this — a few things have shifted since @bensig's 2026-04-12 note:

• #784 (merged 2026-04-13) added file-level fcntl.flock so two processes can't write the same file's drawers at once. That closes one failure mode, but two separate processes each holding their own chromadb.PersistentClient can still corrupt the shared HNSW index — the Rust-side in-memory view doesn't coordinate across processes, which is the same root cause as what you hit here.
• Proposed direction for "I want parallel mining": single process with internal fan-out via ThreadPoolExecutor, not two separate mempalace mine invocations. That keeps everything under one PersistentClient and still respects mine_lock() per file. Filed as #1088 with benchmarks on our fork (~14 min → ~3 min on a 165K-drawer palace). Gated on #1071 (ORT thread cap, MERGEABLE) landing first so workers don't oversubscribe cores.

Short-term recommendation for anyone arriving at this issue: don't run mempalace mine project1 & mempalace mine project2 & wait — that's two PersistentClients and still unsafe as of v3.3.2. If sequential is too slow today, the fork's --workers N implementation (single-process fan-out, jphein/mempalace) has been running on a 165K-drawer palace since 2026-04-10 without the HNSW corruption you documented here.

@bensig — does closing this via "#784 merged + #1088 once it lands" map to how you'd frame the fix path?

[jphein]

Update on the close path I asked about above —

#976 (@felipetruman, merged 2026-04-25) ships mine_global_lock() in palace.py, which is exactly the fan-out-collapse pattern. Two simultaneous mempalace mine invocations now fail-fast on the second with MineAlreadyRunning instead of corrupting HNSW. Combined with the existing per-file mine_lock from #784 and the hnsw:num_threads=1 pin (also in #976) that closes the parallel-insert race even within a single process, the original failure mode here is fully addressed on develop.

The ergonomic question I raised about "I want parallel mining" (#1088 single-process fan-out) is separate and remains open as a perf request, not a correctness one.

@fubak — would you be able to confirm one of these on a fresh palace once v3.3.4 ships (or sooner via pip install git+https://github.com/MemPalace/mempalace.git@develop):

• Two simultaneous mempalace mine calls no longer corrupt HNSW — second one exits cleanly?
• Or, if you've moved on, a 👍 / "no longer affected" is enough to close.

@bensig — happy to revise framing if you'd rather see #976 close this directly.

[fubak]

Confirmed fixed on develop (v3.3.3)

Tested on v3.3.3 (pip install git+https://github.com/MemPalace/mempalace.git@develop).

Two simultaneous mempalace mine calls against the same palace:

MEMPALACE_PALACE_PATH=/tmp/test-palace mempalace mine /tmp/project --wing test &
MEMPALACE_PALACE_PATH=/tmp/test-palace mempalace mine /tmp/project --wing test &
wait

Result:

Mine 1 (the one that lost the lock race):

mempalace: another `mine` is already running against /tmp/test-palace — exiting cleanly.

Mine 2 completed normally. Palace intact — search works post-mine.

No HNSW corruption, no silent failure. The MineAlreadyRunning path works exactly as described. This closes the original failure mode from the issue.

[Zarathustraed]

happens to me too, at the end i have a chat window with some agent whos been fixing this at its own this whole time and catching up on missed mempalaced content. All I'm telling it, is its not working again. Pretty dumb but it works.