earcher.py does not use i18n regex patterns — Japanese (and other non-English) search is degraded

[m2a8-sketch]

searcher.py does not use i18n regex patterns — Japanese (and other non-English) search is degraded

Summary

searcher.py does not reference the i18n module at all. The BM25 tokenizer uses a hardcoded regex (\w{2,}) and does not apply language-specific stop words or topic patterns from i18n/*.json. This means the Japanese stop words, quote patterns, and action patterns defined in ja.json are unused during search, degrading search quality for Japanese content.

Environment

• MemPalace: v3.3.0
• Python: 3.11
• OS: Windows 11
• Palace content: ~2,700 drawers, primarily Japanese conversation logs

Steps to Reproduce

1. Mine Japanese conversation logs with mempalace mine <dir> --mode convos
2. Search for a specific Japanese term: mempalace search "レッサーパンダ"
3. Observe that results include unrelated content with high match scores, while relevant results (containing the exact term) are ranked alongside noise

Expected Behavior

• BM25 should strip Japanese stop words (は, が, を, に, で, etc.) defined in ja.json before scoring
• Exact keyword matches in Japanese should be strongly boosted
• Language-specific regex patterns (topic_pattern, action_pattern) should be used for closet generation and search

Actual Behavior

• searcher.py does not import or call anything from mempalace.i18n
  • Verified with: 'i18n' in text → False, 'load_lang' in text → False
• The tokenizer _TOKEN_RE = re.compile(r"\w{2,}", re.UNICODE) handles Unicode but does not benefit from language-specific stop word removal
• Japanese stop words remain in the token stream, diluting BM25 IDF scores
• i18n/__init__.py defaults to English on import (load_lang("en")) and config.py has no lang property to override this

Suggested Fix

1. Add a lang field to config.json (e.g., "lang": "ja")
2. Have MempalaceConfig expose this field and call load_lang() during initialization
3. In searcher.py, use i18n.get_regex()["stop_words"] to filter tokens in _tokenize() before BM25 scoring
4. Optionally use topic_pattern and action_pattern for closet generation in the relevant modules

Additional Context

The i18n/ja.json file is well-structured and contains useful patterns. The infrastructure is all there — it just needs to be wired into the search pipeline. This would also benefit the other 7 supported languages (fr, ko, es, de, zh-CN, zh-TW, en).

[mvalentsev]

Confirmed in develop:

• searcher.py has 0 imports from mempalace.i18n
• _TOKEN_RE = re.compile(r"\w{2,}", re.UNICODE) at line 30 is language-agnostic
• _tokenize() at lines 47-49 does not filter stop words
• Used in BM25 path (lines 74, 78, 469), so Japanese particles like は/が/を/に/で stay in the token stream

The infrastructure is wired up elsewhere already:

• i18n.get_regex() returns {topic_pattern, stop_words, quote_pattern, action_pattern} at i18n/__init__.py:89
• dialect.py:342, entity_detector.py:36, palace.py:147, closet_llm.py:121 all consume it
• ja.json has the stop_words list populated

searcher.py is one place where this was never wired.

Scope as you described:

1. config.py gets a lang field, could fall back to entity_languages[0] which already exists, or to "en"
2. searcher._tokenize() pulls stop_words from get_regex() and filters them out before BM25

One nuance: i18n/__init__.py:285 calls load_lang("en") at import time and current_lang is module-global. Whoever initializes the config would need to call load_lang(cfg.lang) early, or get_regex could take a lang argument. Worth checking which direction the maintainers prefer.

Related but orthogonal:

• [Feature] Add Multilingual Support  #231, meta multilingual (room classification, memory extraction, spellcheck)
• Feature Request: Improve Chinese semantic search quality #516, Chinese semantic search quality (embedding path)
• feat: make embedding model configurable for non-English search #553 PR, configurable embedding model for non-English (embedding path)

#973 is specifically the BM25/lexical path, so it's separable from the embedding-side work.

[mvalentsev]

Followed up with #977 for the wiring. Quick heads-up on your Japanese setup specifically.

_TOKEN_RE = \w{2,} currently tokenizes "プロジェクトをしました" to a single 13-character token, so the stop-word filter on top has nothing to remove. Japanese, Chinese, and Korean need a word segmenter (MeCab, jieba, konlpy) before the ja.json stop-words list does anything useful. That segmentation work is separate from the filter wiring in #977; adding a CJK optional dep is a bigger design decision that wants maintainer input.

Space-separated locales (en, ru, fr, de, es, pt-br, it, id, hi) get the filter immediately once #977 lands.

[mvalentsev]

Small amendment after a code review pass on #977.

The stop-word filter is now opt-in: it activates only when the user sets MEMPALACE_LANG / MEMPAL_LANG or config.json["lang"]. Palaces that never configured a language keep their pre-PR BM25 scoring byte-for-byte, so the landing is non-breaking for English users who have not explicitly set a locale.

To turn the filter on, set one of the values above to your language code (en, ru, fr, de, es, pt-br, it, id, hi). The CJK segmentation limitation from the earlier note still stands, so setting ja does not gain much without a word segmenter.

[m2a8-sketch]

Thanks for the quick confirmation and for wiring this up in #977, @mvalentsev!

The CJK segmentation point makes total sense — \w{2,} treating an entire Japanese clause as one token is exactly what I'm seeing.

One additional thing I've noticed: searching Japanese text via the MCP server throws TextInputSequence must be str, while the same query works fine on the CLI (mempalace search "レッサーパンダ"). This seems to be a separate issue in the MCP path — possibly a type mismatch before the query reaches ChromaDB. Should I open a separate issue for that?

Appreciate the thorough response.

[mvalentsev]

Yes, please open a separate issue. That's clearly a different bug from the stop-word filter, which is purely a scoring-side change and doesn't touch the embedding path.

TextInputSequence must be str is a tokenizers library error that fires when input is bytes or a non-str type by the time it reaches the embedding model. The fact that the identical query string works on the CLI narrows it to something in the MCP request path rather than the search or embedding code itself.

Helpful info to include in the new issue:

• OS + Python version
• Which MCP client (Claude Desktop / Cursor / other)
• Full stack trace
• Whether it reproduces on short ASCII queries or only on non-ASCII / Japanese input
• Exact input that triggers it (escaped unicode vs literal)

If you're on Windows, take a quick look through the open PRs tagged around MCP stdin/stdout encoding. There's at least one in flight on that path.

[m2a8-sketch]

Opened the MCP issue as suggested — see #[1025]. Thanks again for the quick turnaround on #977!