fix(migrate): close SQLite connection and clean temp palace on exception#1216
Open
arnoldwender wants to merge 1 commit intoMemPalace:developfrom
Open
fix(migrate): close SQLite connection and clean temp palace on exception#1216arnoldwender wants to merge 1 commit intoMemPalace:developfrom
arnoldwender wants to merge 1 commit intoMemPalace:developfrom
Conversation
Two resource leaks in mempalace.migrate that bite hardest on Windows: 1. extract_drawers_from_sqlite() opened a sqlite3 connection at the top and only closed it in the happy path. Any exception during query/iteration leaked the handle, leaving a file lock on chroma.sqlite3 that prevented the rest of the migration from touching the palace directory. Wrap with contextlib.closing(). 2. migrate() called tempfile.mkdtemp() and never cleaned it up if a later step (chromadb open, batch import, count, swap) raised. The orphaned palace under the system temp root could be 100s of MB and never gets reclaimed. Wrap the import-and-swap dance in try/finally and rmtree the temp dir if it still exists at the end (os.replace consumes it on the happy path so the existence guard makes that case a no-op). Tests cover the happy-path extraction and verify mkdtemp's directory is removed when ChromaBackend.get_or_create_collection() raises.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and Why
Two resource leaks in
mempalace.migratethat bite hardest on Windows:1. SQLite connection leak in
extract_drawers_from_sqlite()mempalace/migrate.py:55opened a connection at the top of the function and only closed it on the happy path. Any exception during query/iteration leaked the handle. On Windows that holds a file lock onchroma.sqlite3— the rest of the migration cannotshutil.copytreeoros.replacethe palace directory until the process exits. Users see a confusingPermissionErrorafter a partial extraction.2. Orphan temp dir in
migrate()mempalace/migrate.py:232(now ~234) calledtempfile.mkdtemp(prefix="mempalace_migrate_")with notry/finally. If anything between mkdtemp and the finalos.replace(temp_palace, palace_path)raises (chromadb open failure, batch import failure, count mismatch, EXDEV fallback failure), the partial palace directory under the system temp root leaks forever. For a 50K-drawer palace this can be hundreds of MB the user has to find and clean by hand.Fix
extract_drawers_from_sqlite— wrap the connection incontextlib.closing(sqlite3.connect(db_path))soclose()runs on every exit path.detect_chromadb_versionalready had atry/finallyand is left alone.migrate— wrap the temp-palace import-and-swap dance intry/finally. The cleanup checksos.path.exists(temp_palace)first; on the happy pathos.replaceconsumes the temp dir so the guard makes cleanup a no-op.Test plan
test_extract_drawers_returns_drawers— happy path still works after theclosingrefactor (real SQLite fixture with the schemaextract_drawers_from_sqlitereads)test_migrate_cleans_temp_palace_on_chromadb_failure—mkdtemp's directory is removed whenChromaBackend.get_or_create_collectionraises, asserted via atracking_mkdtempthat captures the pathtest_migrate.pytests passtest_backends.py::test_pin_hnsw_threads*failures unrelated — chromadbconfiguration_json["hnsw"]schema drift, also fails on cleandevelop)