Conversation
`col.get(limit=total)` causes SQLite "too many SQL variables" on palaces with >10k drawers (#802) and on older versions the hardcoded limit=10000 silently truncated the count (#850). Paginate in 5k batches using offset and aggregate wing/room counts incrementally. Also use `col.count()` for the header instead of `len(metas)` so the displayed total is always correct. Tested on a 122,686-drawer palace. Fixes #850 Related: #802, #723
zh-TW and zh-CN previously had no `entity` section. Calling
`detect_entities(..., languages=("zh-TW",))` silently fell back to
English patterns (i18n/__init__.py:231-233), so no Chinese names
were ever extracted — Chinese-speaking users got zero people or
projects detected from their own notes.
This adds entity sections for both locales:
- `candidate_pattern`: common-surname-prefixed CJK n-grams (~100
surnames covering >95% of Taiwanese / PRC names), length capped
at {1,2} trailing chars so greedy matches don't swallow the
trailing verb character (e.g. 朱宜振說).
- `boundary_chars`: `\u4E00-\u9FFF` so the i18n loader's
script-aware wrap (introduced in #932) fires `\b` at CJK↔non-CJK
transitions. This is the same mechanism used for Devanagari,
applied to the CJK range.
- `person_verb_patterns`: Chinese verbs attach directly to the
name with no whitespace, so patterns are written as `{name}說`,
`{name}問`, `{name}決定` — no `\b` or `\s+` separators.
- `dialogue_patterns`: full-width colon `：`, Chinese quotes
「」『』, plus the standard Latin forms.
- `pronoun_patterns`: 他 / 她 / 它 / 他們 / 她們 / 您 / 咱.
- `stopwords`: ~140 common particles, pronouns, time expressions,
question words, conjunctions, UI nouns, and politeness forms.
**Known limitation** (explicitly covered by a test): CJK scripts
have no word delimiters, so a name flanked by CJK on both sides
with no punctuation or whitespace break is not extracted. This
is a fundamental limit of regex-based CJK entity detection —
resolving it would require a dictionary tokeniser. Realistic
Chinese technical writing contains enough non-CJK neighbours
(bullet lines, inline English, full-width punctuation, newlines)
that 3+ occurrences normally produce matches. Verified against a
realistic zh-TW PKM note: 朱宜振 extracted 11x from 8 sentences
with 0.99 person-classification confidence.
**Follow-ups** (separate PRs): same pattern for `ja` and `ko`,
both of which currently share the silent fallback-to-English bug.
Tests: 7 new tests in `tests/test_entity_detector.py`:
- `test_zh_tw_candidate_extraction_at_boundaries`
- `test_zh_tw_person_classification`
- `test_zh_tw_stopwords_filter_common_particles`
- `test_zh_tw_falls_back_to_english_for_non_cjk_names`
- `test_zh_cn_candidate_extraction`
- `test_zh_cn_and_zh_tw_union_covers_both_variants`
- `test_zh_tw_known_limitation_inline_name_no_boundary`
Full suite: 957 passed, 0 failed.
build_graph() scans every drawer's metadata in 1000-item batches on every call — O(n) per graph build with no caching. At 50K+ drawers this costs several seconds per MCP tool call (traverse, find_tunnels, graph_stats all call build_graph on every invocation). Add a module-level cache (nodes + edges + timestamp) with a 60-second TTL. Cache is invalidated via invalidate_graph_cache(), exported for write operations to call. Tests updated with setup_method cache resets and two new tests verifying cache hit and invalidation behaviour.
Addresses Copilot review feedback on #661. Co-Authored-By: Claude Opus 4.6 <[email protected]>
Address review feedback from @bensig: 1. Wrap cache reads/writes in threading.Lock for thread safety 2. Promote the col-arg caveat from inline comment to docstring Co-Authored-By: Claude Opus 4.6 <[email protected]>
Collapse implicit string concatenation to single-line strings to satisfy ruff format --check in CI. Co-Authored-By: Claude <[email protected]>
Claude Code 2.1.114 passes stop_hook_active:true on every Stop fire
after the first in a session (plugin-dispatched hooks in particular).
The legacy guard at line 426 was written for block-mode, where a
re-fire with the flag set meant "you already blocked, don't block
again" — correct loop prevention when the hook returns
{"decision":"block"}.
Silent-save mode (default since #673) never blocks — it saves
directly and returns. The flag is meaningless there, so the old
guard was suppressing every auto-save after the first one in a
Claude Code session. Symptom: terminal never shows the "✦ N
memories woven" notification again, hook.log stays silent, save
marker stuck.
Fix: only skip on stop_hook_active when block mode is configured.
Silent mode runs through as normal — the save is deterministic and
idempotent, no loop risk.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…rect
mempalace.mcp_server redirects stdout → stderr at module-level import
(both Python-level and fd-level via os.dup2) to protect the MCP stdio
protocol from ChromaDB's C-level noise. Silent-save imports mcp_server
transitively via _save_diary_direct, so by the time _output() calls
print(), sys.stdout is actually stderr.
Claude Code reads hook output from fd 1. With the redirect in effect,
fd 1 points to fd 2, so our {"systemMessage": "✦ N memories woven..."}
JSON lands on stderr and Claude Code never renders it. The save still
happens, the marker still advances — the user just never sees the
beautiful checkpoint notification in their terminal.
Fix: _output() now writes to _REAL_STDOUT_FD (saved by mcp_server before
the redirect) via os.write(), falling back to sys.stdout only when the
saved fd is unavailable (e.g., hooks_cli imported without mcp_server).
Test: bash hook script 2>/dev/null now shows only the JSON;
2>&1 >/dev/null shows only the Diary entry log line — clean separation
restored.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- _output(): use sys.modules.get() instead of unconditional import to avoid triggering mcp_server's stdout redirect as a side effect - _output(): write-all loop for os.write() to handle partial writes and EINTR; fall back to sys.stdout.buffer on OSError - _output() docstring: remove inaccurate _save_diary_direct reference - stop_hook_active guard: narrow except to ImportError/AttributeError, default silent_guard=False (safe: preserves block-mode loop prevention when config load fails) and log a warning instead of silently changing behavior - tests: two new regression tests covering the real-stdout-fd path and the fd-1 fallback path Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Upstream develop commit feba7e8 (2026-04-18) added `m = m or {}` to the single-shot `for m in metas:` loop after this branch already rewrote status() to paginate. Without porting the guard forward, merging this PR would silently drop jp's fix and crash again on palaces with null-metadata drawers. Addresses bensig's review on #851.
…ate-large-palaces # Conflicts: # mempalace/miner.py
…suppress saves Addresses bensig's review on PR #1021. silent_guard was initialized to False, so when both MempalaceConfig import and .hook_silent_save attribute access failed, silent_guard stayed False. Then `if not silent_guard:` fired and returned empty — silently dropping the save. In silent mode (the default since v3.3.0), saves should ALWAYS proceed on config-read failure. Changing the initial value to True makes that the safe default. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
feat(i18n): add Traditional + Simplified Chinese entity detection
feat(i18n): add entity detection to German, Spanish, and French locales
feat(i18n): add Belarusian
fix(hooks): resolve Claude plugin hook runner cross-platform
The MCP server config used `python -m mempalace.mcp_server` which fails when mempalace is installed via pipx or uv, since the system python cannot find the module in the isolated venv. Adding a `mempalace-mcp` console_scripts entry point ensures the MCP server works regardless of installation method (pip, pipx, uv, conda). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Takes the candidate set produced by phase-1 detection (manifests, git authors, regex on prose) and asks an LLM to reclassify each candidate as PERSON / PROJECT / TOPIC / COMMON_WORD / AMBIGUOUS. Scale approach: never feed the raw corpus to the LLM. For each candidate, collect up to 3 context lines from sampled prose, cap each at 240 chars, batch 25 candidates per call. Keeps total input around 50-100K tokens even on large corpora and completes in a few minutes on a 4B local model. Interactive UX: - Stderr progress bar with the current candidate name, updates per-batch. - Ctrl-C interrupts cleanly: returns a RefineResult with `cancelled=True` and whatever was classified before the interrupt. The partial result is safe to pass straight to confirm_entities. - Per-batch errors (transport, parse) are recorded in `errors` and don't abort the whole run. Refinement scope: only `uncertain` and low-confidence `projects` entries are sent. Manifest-backed projects (conf >= 0.95) and git- authored people are already authoritative and skip the LLM. Response parser is defensive — accepts `label` or `type` keys, lowercase/uppercase variants, top-level list or wrapped object, and strips markdown code fences. Unknown labels become AMBIGUOUS so the user reviews them rather than silently accepting a bad classification. `collect_corpus_text` provides a simple stratified prose sampler (recent first, capped per-file) so callers don't need to build their own corpus window. 28 tests with a FakeProvider (no network). Covers context collection, prompt building, response parsing variants, classification apply, end-to-end refine, and Ctrl-C partial-result behavior.
Extends the init orchestrator to consume two new signal sources:
1. Claude Code conversation dirs: when the target is a
`~/.claude/projects/` root, convo_scanner contributes ProjectInfo
entries alongside the git/manifest projects. Dedup is by name,
preferring the entry with more user-authored activity.
2. Optional LLM refinement: when --llm is passed, discover_entities
constructs the provider, validates availability, and runs
llm_refine.refine_entities on the merged candidates. Status
summary (reclassified / dropped / cancelled / batch errors)
prints to stderr.
New init flags (opt-in, default remains zero-API):
- --llm: enable refinement
- --llm-provider: ollama (default) | openai-compat | anthropic
- --llm-model: default gemma4:e4b for Ollama
- --llm-endpoint: URL (required for openai-compat)
- --llm-api-key: falls back to env ($ANTHROPIC_API_KEY or
$OPENAI_API_KEY depending on provider)
Provider check_available runs before the scan, so the user sees an
immediate error ("Run: ollama pull <model>" or "ANTHROPIC_API_KEY not
set") rather than a mid-scan failure.
Agent-Logs-Url: https://github.com/MemPalace/mempalace/sessions/3c277c46-20b3-4a43-8eb7-8ee2eb3cb55a Co-authored-by: igorls <[email protected]>
Agent-Logs-Url: https://github.com/MemPalace/mempalace/sessions/3c277c46-20b3-4a43-8eb7-8ee2eb3cb55a Co-authored-by: igorls <[email protected]>
Agent-Logs-Url: https://github.com/MemPalace/mempalace/sessions/3c277c46-20b3-4a43-8eb7-8ee2eb3cb55a Co-authored-by: igorls <[email protected]>
Agent-Logs-Url: https://github.com/MemPalace/mempalace/sessions/3c277c46-20b3-4a43-8eb7-8ee2eb3cb55a Co-authored-by: igorls <[email protected]>
Agent-Logs-Url: https://github.com/MemPalace/mempalace/sessions/3c277c46-20b3-4a43-8eb7-8ee2eb3cb55a Co-authored-by: igorls <[email protected]>
…oritative sources Addresses issues found while reviewing the initial phase-2 implementation against real data: **Bug: uncertain bucket starved from the LLM.** `discover_entities` was dropping the regex-uncertain bucket whenever real git/manifest signal existed — which is exactly when `--llm` is most useful for cleaning up prose noise. The uncertain candidates never reached the refinement step. Fixed: only drop when `llm_provider is None`. **Context collection: word boundaries, not substring.** `_collect_contexts` used substring matching on lower-cased lines, so the name "Go" matched "good", "going", "forgot". Switched to a `(?<!\w)…(?!\w)` regex so short names only match at token boundaries. **Authoritative-source detection replaces confidence threshold.** Previously the refinement step skipped entries with `confidence >= 0.95` to avoid second-guessing manifest-backed projects. That threshold was fragile — the regex detector produces 0.99 confidence for things like `code file reference (5x)` on framework names (OpenAPI, etc.), so those skipped the LLM despite being regex-only noise. New helpers `_is_authoritative_person` / `_is_authoritative_project` look at the actual signal strings (commits, package.json, etc.) to decide. **Now also refines regex-derived people.** After #1148's high-pronoun-signal fix, the regex detector can promote non-people to the `people` bucket (e.g. a capitalized common noun that happened to appear near pronouns). The LLM now gets a chance to clean those up, while git-authored people are still skipped. **Robust JSON extraction.** Small local models routinely wrap JSON output in prose ("Sure, here's the classification: {…}"). The previous code-fence stripper failed on that. `_extract_json_candidates` now does balanced-bracket extraction with string-aware quote handling, so it recovers JSON from: - raw responses - markdown fenced blocks - JSON embedded inside surrounding text - multiple candidate objects/arrays **Prompt guidance for frameworks vs user projects.** Added an explicit instruction: frameworks, runtimes, APIs, cloud services, and third-party vendors (Angular, OpenAPI, Terraform, Bun, Google, etc.) are TOPIC unless the context clearly says it's the user's own codebase. Directly addresses a false-positive pattern observed during dev runs. **Defensive mtime.** `convo_scanner._safe_mtime` catches OSError during `stat()` — permission changes, filesystem races, broken symlinks — and sorts the affected file to the end of the newest-first order rather than crashing the scan. **Cosmetic:** merged two adjacent f-strings on the same line in `backends/chroma.py` and `llm_client.py` (no behaviour change). 15 new tests cover the OSError fallback, word-boundary matching, JSON extraction variants, authoritative-source helpers, refining high- confidence regex projects, and end-to-end LLM refinement preserving the uncertain bucket.
…egistry
The init step's output was a dead file. miner.py has always read
`~/.mempalace/known_entities.json` to tag drawer metadata with
recognized names, but nothing ever wrote it — so init's careful
manifest + git + LLM detection work stopped at `<project>/entities.json`
and never reached the path that actually uses it.
Measured delta on a representative prose snippet (eight sentences
mentioning six real people and four real projects):
- Empty registry: 0 entities recognized (multi-word names fail the
frequency threshold; lowercase/hyphenated project names don't match
the CamelCase regex).
- Registry populated by init: 12 entities recognized (all correct, zero
false positives).
Every recognized name becomes a semicolon-separated metadata tag on the
drawer, which ChromaDB uses for entity-filtered search.
Implementation:
- `miner.add_to_known_entities({category: [names]})` reads the existing
registry, unions each category (case-insensitively, preserving first-
seen casing), and writes back. The function is tolerant of the two
on-disk shapes miner already supports: list of names, or dict mapping
name → code (dialect-style). In the dict case new names are added as
keys with `None` values so existing codes aren't overwritten.
- Invalidates the in-process mtime cache so same-process callers
(`cmd_init` → `cmd_mine` in one run) see the write immediately.
- Writes with `ensure_ascii=False` so non-ASCII names (Gergő Móricz,
Arturo Domínguez, etc.) stay readable on disk.
- Chmods 0o600 — the registry mirrors confirm-step PII from the user's
git authors and local paths.
cmd_init now calls this at the end of the confirm-entities step, after
the per-project `entities.json` is written (which is kept as an audit
trail the user can inspect or hand-edit). The per-project file is still
excluded from mining via `SKIP_FILENAMES` from the earlier fix.
17 new tests cover: fresh-file creation, list-category union, case-
insensitive dedup, preservation of untouched categories, dict-format
registries, malformed/non-dict file recovery, cache invalidation,
unicode round-trip, and an end-to-end verification that the miner's
`_extract_entities_for_metadata` picks up every registered name.
Agent-Logs-Url: https://github.com/MemPalace/mempalace/sessions/76794fde-2383-4674-ab36-f89ad803eeb2 Co-authored-by: igorls <[email protected]>
fix(3.3.3): two followups from #1145 before tag cut
…xpanduser MEMPALACE_PALACE_PATH (and legacy MEMPAL_PALACE_PATH) read from the environment was returned as-is from Config.palace_path, while the sibling --palace CLI path gets os.path.abspath() applied at mcp_server.py:62. That inconsistency means env-var callers can end up with literal '~' or unresolved '..' segments in the path, which (a) breaks user intuition and (b) lets a caller who can set env vars on the target user's session redirect palace storage to an unexpected location. Apply os.path.abspath(os.path.expanduser(...)) to the env-var branch so both code paths converge on the same resolved absolute path. Closes #1163
The new abspath+expanduser normalization means /env/palace no longer round-trips literally on Windows (abspath prepends the current drive, producing D:\env\palace). Rewrite the env-var tests to compare against os.path.abspath(os.path.expanduser(raw)) instead of hardcoded Unix strings, and build raw paths with os.path.join so backslash-vs-slash differences don't leak into assertions. Covers test_env_override, the three new tests, and the legacy-alias test in test_config_extra.
Windows 8.3 short paths legitimately contain tildes (e.g. the CI runner's USERPROFILE resolves to C:\Users\RUNNER~1\...), so asserting "~" is absent from the expanded path fails on Windows even when expanduser worked correctly. The equality check against os.path.abspath(os.path.expanduser()) is authoritative; drop the redundant absence heuristic.
…etection feat(init): scan manifests and git authors for real entity signal (v1)
feat(init): wire confirmed entities into the miner's known-entities registry
#1148, #1150, and #1157 were reviewed and merged on GitHub, but the two stacked children landed on their parent feature branches (now stale) rather than on develop. Only #1148's commits reached develop via the direct merge. Release PR #1159 (develop → main for v3.3.3) is therefore missing the LLM refinement, Claude-conversation scanner, and miner- registry wire-up that were ostensibly part of the release. This merge brings the stale `feat/llm-entity-refine` branch (which contains the rolled-up merge commit for #1157 → #1150 → everything below) into develop so the release tag includes it. No code changes here — only history recovery.
3 tasks
…ources `discover_entities` was deduping the convo_scanner results against the manifest/git scan with a case-sensitive key, while every other dedup path in the pipeline (`_merge_detected`, `miner.add_to_known_entities`) uses case-insensitive matching. A project named `foo` in a manifest plus `Foo` as a Claude Code `cwd` variant would surface as two review entries instead of collapsing to one. Fix keys `by_name` by `name.lower()` while preserving the first-seen casing, matching the rest of the pipeline. Flagged by Copilot on #1175. Regression test asserts a manifest project + a CamelCase-variant convo cwd for the same real project collapse to one entry.
…nv-normalize fix(security): normalize MEMPALACE_PALACE_PATH env var with abspath+expanduser
Adds entries to the 3.3.3 section for the work that landed via #1148, #1150, #1157, and #1175 (rescued from stacked feature branches into develop via #1175). Without these entries the 3.3.3 release notes on main would advertise only the hook/diary/search fixes that made it to develop through the first direct merge. Covers: - Manifest + git-author entity detection (#1148) - Regex detector accuracy improvements (#1148) - Optional --llm classification with Ollama / openai-compat / Anthropic provider abstraction and interactive UX (#1150) - Claude Code conversation scanner (#1150) - Init → miner registry wire-up so confirmed entities actually reach drawer metadata tagging (#1157) - Case-insensitive project dedup across all sources (#1175) - `mempalace mine` skips the generated entities.json artifact
3 tasks
Co-authored-by: Copilot <[email protected]>
…rhaul docs(changelog): document init entity-detection overhaul in 3.3.3
igorls
approved these changes
Apr 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release merge for v3.3.3. Fast-forwards `main` from v3.3.1 → v3.3.3 content, unblocking the tag cut.
What's in this merge
53 commits on `develop` since v3.3.1 — everything that shipped in v3.3.2 plus the v3.3.3 delta.
The v3.3.3 fix — the reason we're cutting now
New in v3.3.3 (since v3.3.2 tag, 2026-04-20)
Bug fixes
mempalace-mcpentry point missing frompyproject.tomldespiteplugin.jsonrequiring it — MCP server fails to start on fresh install #1093 fix)Improvements
Added
Already-landed (shipped in v3.3.2, merged here because `main` lagged)
All 20+ PRs in the [3.3.2] CHANGELOG section.
Deferred to v3.3.4
Release-readiness
Post-merge sequence
```bash
git checkout main && git pull
git tag -a v3.3.3 -m "v3.3.3 — restore install integrity"
git push origin v3.3.3
then: python -m build && twine upload dist/* (manual, needs PyPI token)
```
Notes
Smoke-test + brief: go-team-mempalace/proposals/v3-3-3-igor-brief.html.
Ready to merge.