proposal: concurrent mining via ThreadPoolExecutor + `bulk_check_mined()` pre-fetch

[jphein]

Context

On our fork's palace (165,632 drawers, mixed project files + conversation transcripts), a cold-run single-threaded mempalace mine takes ~14 min. Two changes bring it to ~3 min on a 4-core machine without changing any other semantics:

1. bulk_check_mined() — paginated pre-fetch of (source_file, mtime) pairs in 10 K batches, so the "is this file already mined?" check doesn't do a per-file ChromaDB query.
2. --workers N flag on mempalace mine — a ThreadPoolExecutor(max_workers=N) fans out process_file() across not-yet-mined files.

Both have been running in production on jphein/mempalace since 2026-04-10. Per-file correctness is preserved because process_file() still acquires mine_lock() per file (from #784) before writing, so the fan-out never races on the same file's drawers.

Why an issue rather than a PR

--workers overlaps in intent with #784 (file-level locking, merged 2026-04-13) — both care about safe concurrency during mining — but the mechanisms differ:

• fix: file-level locking to prevent multi-agent duplicate drawers #784 — per-file fcntl.flock prevents two processes writing the same file's drawers concurrently.
• This proposal — a single process fanning out across different files, still respecting fix: file-level locking to prevent multi-agent duplicate drawers #784's per-file lock on each.

The question is whether the maintainer sees this as a natural extension of #784's concurrency story, or prefers a different direction (e.g., multi-process orchestration outside a single mine invocation). Would rather ask than file a 200-line PR that goes the wrong way.

Concrete numbers

Run environment: MacBook M2, Python 3.13, chromadb 1.5.8, 165 K-drawer palace (~8 K unique source files).

Config	Wall time
Single-threaded (upstream current)	~14 min
bulk_check_mined() pre-fetch only	~8 min
--workers 4 + bulk_check_mined()	~3 min
--workers 8 + bulk_check_mined()	~2 min 50 s (diminishing returns past 4)

Open questions

1. Is in-process fan-out with ThreadPoolExecutor of interest upstream, or is "multiple mempalace mine invocations handling disjoint subsets" the preferred concurrency model?
2. Should --workers default to 1 (current single-threaded behavior) with explicit opt-in, or to min(4, cpu_count())?
3. Any concern about bulk_check_mined() memory footprint for palaces with O(100 K+) unique source files? (On our 165 K-drawer palace with ~8 K unique files, the pre-fetch is <1 MB. An O(500 K)-file palace would be ~4 MB — still fine, but a chunked iterator might be warranted at extreme scales.)

Happy to open a PR immediately if the direction is approved. If not, close this with a note and we'll keep it fork-local.

Code for reference (fork main):

• mempalace/palace.py — bulk_check_mined()
• mempalace/miner.py — concurrent mine() with --workers

[jphein]

Cross-references I missed on the first pass — adjacent upstream work changes the framing of this proposal:

• Parallel mining corrupts ChromaDB HNSW index — no warning, silent failure #357 (@fubak, 2026-04-13) — "Parallel mining corrupts ChromaDB HNSW index, no warning, silent failure." This is the exact failure mode fix: file-level locking to prevent multi-agent duplicate drawers #784's file-locking was built to prevent. It's also the strongest argument for this proposal's direction: --workers as a single-process, lock-respecting fan-out is a safer way to get parallelism than external orchestration that can race the lock (which is what Parallel mining corrupts ChromaDB HNSW index — no warning, silent failure #357 documents). If this lands with process_file() still acquiring mine_lock() per file, it should close Parallel mining corrupts ChromaDB HNSW index — no warning, silent failure #357 cleanly.
• Background mempalace mine pins 400–500 % CPU — ORT intra_op pool ignores OMP env vars #1068 (@sha2fiddy, 2026-04-21) / fix: cap ONNX intra_op threads via MEMPAL_MAX_THREADS #1071 (PR +391/-5, MERGEABLE) — report + fix for the orthogonal problem that ONNX Runtime's intra_op pool ignores OMP_NUM_THREADS / MKL_NUM_THREADS / OPENBLAS_NUM_THREADS, so concurrent mining pins every core regardless of worker count. This matters here because --workers 4 with uncapped ORT pools is effectively 4 × (all cores), not 4 × (1 core). For --workers to be honest about the parallelism it grants, it has to compose correctly with MEMPAL_MAX_THREADS from fix: cap ONNX intra_op threads via MEMPAL_MAX_THREADS #1071 — e.g., --workers 4 MEMPAL_MAX_THREADS=2 should be 8 total threads, not 4 × unbounded.
• SIGSEGV in HNSW parallel inserts — missing hnsw:num_threads on collection creation #974 (@vince-mod3) — SIGSEGV in HNSW parallel inserts, addressed by the hnsw:num_threads cap in fix: cap ONNX intra_op threads via MEMPAL_MAX_THREADS #1071.

Revised ordering: #1071 should land first (gives --workers a bounded-thread baseline to reason about), then this proposal can be filed as a PR against a cleaner substrate, framed as closing #357 rather than a speculative concurrency extension.

Thanks @fubak for the #357 repro — should have found that before filing here. Leaving this issue open for direction-check, but no PR until #1071 lands.

[jphein]

For reviewer context — this proposal is orthogonal to @rboarescu's palace-daemon, which solves a different slice of the concurrency space:

• palace-daemon — FastAPI gateway sitting between multiple clients and one MemPalace. Three asyncio semaphores (read / write / mine) with a dedicated mine lane so bulk imports never starve interactive queries. Includes MCP-over-HTTP proxy for remote machines. Solves the multi-client coordination problem.
• This proposal — single-process in-process parallelism via ThreadPoolExecutor, with mine_lock() per file for safety. Solves the single-client throughput on a large cold-mine problem.

They compose cleanly: palace-daemon handles traffic between clients; this proposal handles how one client finishes its own mine 5× faster. Running our fork's --workers N implementation behind palace-daemon would give both.

One pattern from palace-daemon worth borrowing for future iteration: a two-lane worker split — N-1 mine workers + 1 interactive reserve — so an in-progress mine can't block an ad-hoc search from the same process. Doesn't belong in the initial PR (YAGNI for cold-mine benchmarks where there's no concurrent interactive traffic), but a useful follow-up direction if the queue sees mixed workloads.