Calculating SHA256 of large wheels from HTTP sources causes "memoryview is too large"

[JPK64]

Description

We are having difficulties with package resolution in our project.

We are using the ROCm version of torch in our project, adding https://download.pytorch.org/whl/rocm7.1 as a source. Running poetry lock to generate a lock file (or sometimes even just running poetry install with a lock file already present) causes a ValueError: memoryview is too large. This does not happen when running with --no-cache. Clearing the cache does not fix the issue, however.

I have provided a minimal pyproject.toml below that can be used to reproduce the issue.

From further investigation, this happens while the SHA256 of the wheel is calculated. The wheel is not in the cache yet and thus is downloaded. Reading the response body then causes cachecontrol to use msgpack for serializing the HTTP response. As some wheels are larger than 2 GiB (for example torch-2.11.0+rocm7.1-cp310-cp310-manylinux_2_28_x86_64.whl, which is 5.4 GiB), this likely causes the aforementioned issue in msgpack as the response body probably is too large.

Workarounds

Installing or creating the lock file with --no-cache avoids this issue.

Poetry Installation Method

system package manager (eg: dnf, apt etc.)

Operating System

Arch Linux in WSL

Poetry Version

Poetry (version 2.3.2)

Poetry Configuration

cache-dir = "~/.cache/pypoetry"
data-dir = "~/.local/share/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = false
keyring.enabled = true
python.installation-dir = "{data-dir}/python"  # ~/.local/share/pypoetry/python
repositories.torch-rocm.url = "https://download.pytorch.org/whl/rocm7.1"
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = true
virtualenvs.path = "{cache-dir}/virtualenvs"  # ~/.cache/pypoetry/virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false

Python Sysconfig

Sysconfig is provided here (GitHub wouldn't let me attach a file): https://gist.github.com/JPK64/404cbf7ab93a76a5b6144b9cda0012e5

Example pyproject.toml

[project]
name = "memoryview-issue"
version = "0.1.0"
requires-python = ">=3.14,<3.15"

[tool.poetry.dependencies]
torch = { version = "2.11.0+rocm7.1", source = "torch-rocm" }
triton-rocm = { version = "3.6.0", source = "torch-rocm" }

[[tool.poetry.source]]
name = "torch-rocm"
url = "https://download.pytorch.org/whl/rocm7.1"
priority = "explicit"

[build-system]
requires = ["poetry-core>=2.2.1"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

poetry-runtime.log

Loading configuration file ~/.config/pypoetry/config.toml
Adding repository torch-rocm (https://download.pytorch.org/whl/rocm7.1) and setting it as explicit
Checking keyring availability: Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Backend 'fail Keyring' is not suitable
No valid keyring backend was found
Unavailable
Updating dependencies
Resolving dependencies...
 1: fact: memoryview-issue is 0.1.0
 1: derived: memoryview-issue
 1: fact: memoryview-issue depends on torch (>=2.10,<3.0)
 1: selecting memoryview-issue (0.1.0)
 1: derived: torch (>=2.10,<3.0)
Creating new session for download.pytorch.org
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): download.pytorch.org:443
[urllib3:urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/rocm7.1/torch/ HTTP/1.1" 200 22800
Source (torch-rocm): 2 packages found for torch >=2.10,<3.0
Source (torch-rocm): Downloading: https://download.pytorch.org/whl/rocm7.1/torch-2.11.0%2Brocm7.1-cp310-cp310-manylinux_2_28_x86_64.whl
[urllib3:urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/rocm7.1/torch-2.11.0%2Brocm7.1-cp310-cp310-manylinux_2_28_x86_64.whl HTTP/1.1" 200 5791887404
Source (torch-rocm): Downloading: https://download.pytorch.org/whl/rocm7.1/torch-2.11.0%2Brocm7.1-cp310-cp310-manylinux_2_28_x86_64.whl
[urllib3:urllib3.connectionpool] Resetting dropped connection: download.pytorch.org
[urllib3:urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/rocm7.1/torch-2.11.0%2Brocm7.1-cp310-cp310-manylinux_2_28_x86_64.whl HTTP/1.1" 200 5791887404
Source (torch-rocm): Downloading: https://download.pytorch.org/whl/rocm7.1/torch-2.11.0%2Brocm7.1-cp310-cp310-manylinux_2_28_x86_64.whl
[urllib3:urllib3.connectionpool] Resetting dropped connection: download.pytorch.org
[urllib3:urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/rocm7.1/torch-2.11.0%2Brocm7.1-cp310-cp310-manylinux_2_28_x86_64.whl HTTP/1.1" 200 5791887404
 1: Version solving took 1808.054 seconds.
 1: Tried 1 solutions.

ValueError

list.index(x): x not in list

at /usr/lib/python3.14/site-packages/poetry/repositories/legacy_repository.py:63 in package
     59│         Note that this will be cached so the subsequent operations
     60│         should be much faster.
     61│         """
     62│         try:
  →  63│             index = self._packages.index(Package(name, version))
     64│ 
     65│             return self._packages[index]
     66│         except ValueError:
     67│             package = super().package(name, version)

The following error occurred when trying to handle this error:


Stack trace:

51  /usr/lib/python3.14/site-packages/cleo/application.py:327 in run
     325│ 
     326│             try:
   → 327│                 exit_code = self._run(io)
     328│             except BrokenPipeError:
     329│                 # If we are piped to another process, it may close early and send a

50  /usr/lib/python3.14/site-packages/poetry/console/application.py:260 in _run
     258│ 
     259│             try:
   → 260│                 exit_code = super()._run(io)
     261│             except PoetryRuntimeError as e:
     262│                 io.write_error_line("")

49  /usr/lib/python3.14/site-packages/cleo/application.py:431 in _run
     429│             io.input.interactive(interactive)
     430│ 
   → 431│         exit_code = self._run_command(command, io)
     432│         self._running_command = None
     433│ 

48  /usr/lib/python3.14/site-packages/cleo/application.py:473 in _run_command
     471│ 
     472│         if error is not None:
   → 473│             raise error
     474│ 
     475│         return terminate_event.exit_code

47  /usr/lib/python3.14/site-packages/cleo/application.py:457 in _run_command
     455│ 
     456│             if command_event.command_should_run():
   → 457│                 exit_code = command.run(io)
     458│             else:
     459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

46  /usr/lib/python3.14/site-packages/cleo/commands/base_command.py:117 in run
     115│         io.input.validate()
     116│ 
   → 117│         return self.execute(io) or 0
     118│ 
     119│     def merge_application_definition(self, merge_args: bool = True) -> None:

45  /usr/lib/python3.14/site-packages/poetry/console/commands/installer_command.py:39 in execute
      37│     def execute(self, io: IO) -> int:
      38│         PoetryKeyring.preflight_check(io, self.poetry.config)
   →  39│         return super().execute(io)
      40│ 

44  /usr/lib/python3.14/site-packages/cleo/commands/command.py:61 in execute
      59│ 
      60│         try:
   →  61│             return self.handle()
      62│         except KeyboardInterrupt:
      63│             return 1

43  /usr/lib/python3.14/site-packages/poetry/console/commands/lock.py:44 in handle
      42│         self.installer.lock(update=self.option("regenerate"))
      43│ 
   →  44│         return self.installer.run()
      45│ 

42  /usr/lib/python3.14/site-packages/poetry/installation/installer.py:112 in run
     110│             self.verbose(True)
     111│ 
   → 112│         return self._do_install()
     113│ 
     114│     def dry_run(self, dry_run: bool = True) -> Installer:

41  /usr/lib/python3.14/site-packages/poetry/installation/installer.py:250 in _do_install
     248│                 source_root=self._env.path.joinpath("src")
     249│             ):
   → 250│                 solved_packages = solver.solve(
     251│                     use_latest=self._whitelist
     252│                 ).get_solved_packages()

40  /usr/lib/python3.14/site-packages/poetry/puzzle/solver.py:91 in solve
      89│         with self._progress(), self._provider.use_latest_for(use_latest or []):
      90│             start = time.time()
   →  91│             packages = self._solve()
      92│             # simplify markers by removing redundant information
      93│             for transitive_info in packages.values():

39  /usr/lib/python3.14/site-packages/poetry/puzzle/solver.py:174 in _solve
     172│ 
     173│         try:
   → 174│             result = resolve_version(self._package, self._provider)
     175│ 
     176│             packages = result.packages

38  /usr/lib/python3.14/site-packages/poetry/mixology/__init__.py:18 in resolve_version
      16│     solver = VersionSolver(root, provider)
      17│ 
   →  18│     return solver.solve()
      19│ 

37  /usr/lib/python3.14/site-packages/poetry/mixology/version_solver.py:189 in solve
     187│             while next is not None:
     188│                 self._propagate(next)
   → 189│                 next = self._choose_package_version()
     190│ 
     191│             return self._result()

36  /usr/lib/python3.14/site-packages/poetry/mixology/version_solver.py:596 in _choose_package_version
     594│             package = locked
     595│ 
   → 596│         package = self._provider.complete_package(package)
     597│ 
     598│         conflict = False

35  /usr/lib/python3.14/site-packages/poetry/puzzle/provider.py:491 in complete_package
      489│                 )
      490│             else:
   →  491│                 pool_package = self.get_package_from_pool(
      492│                     package.pretty_name,
      493│                     package.version,

34  /usr/lib/python3.14/site-packages/poetry/repositories/repository_pool.py:159 in package
     157│     ) -> Package:
     158│         if repository_name:
   → 159│             return self.repository(repository_name).package(name, version)
     160│ 
     161│         for repo in self.repositories:

33  /usr/lib/python3.14/site-packages/poetry/repositories/legacy_repository.py:67 in package
      65│             return self._packages[index]
      66│         except ValueError:
   →  67│             package = super().package(name, version)
      68│             package._source_type = "legacy"
      69│             package._source_url = self._url

32  /usr/lib/python3.14/site-packages/poetry/repositories/cached_repository.py:70 in package
      68│ 
      69│     def package(self, name: str, version: Version) -> Package:
   →  70│         return self.get_release_info(canonicalize_name(name), version).to_package(
      71│             name=name
      72│         )

31  /usr/lib/python3.14/site-packages/poetry/repositories/cached_repository.py:52 in get_release_info
      50│             return PackageInfo.load(self._get_release_info(name, version))
      51│ 
   →  52│         cached = self._release_cache.remember(
      53│             f"{name}:{version}", lambda: self._get_release_info(name, version)
      54│         )

30  /usr/lib/python3.14/site-packages/poetry/utils/cache.py:147 in remember
     145│         value = self.get(key)
     146│         if value is None:
   → 147│             value = callback() if callable(callback) else callback
     148│             self.put(key, value, minutes)
     149│         return value

29  /usr/lib/python3.14/site-packages/poetry/repositories/cached_repository.py:53 in <lambda>
      51│ 
      52│         cached = self._release_cache.remember(
   →  53│             f"{name}:{version}", lambda: self._get_release_info(name, version)
      54│         )
      55│ 

28  /usr/lib/python3.14/site-packages/poetry/repositories/legacy_repository.py:120 in _get_release_info
     118│         yanked = page.yanked(name, version)
     119│ 
   → 120│         return self._links_to_data(
     121│             links,
     122│             PackageInfo(

27  /usr/lib/python3.14/site-packages/poetry/repositories/http_repository.py:367 in _links_to_data
     365│                     break
     366│             else:
   → 367│                 file_hash = self.calculate_sha256(link)
     368│ 
     369│             if file_hash is None and (

26  /usr/lib/python3.14/site-packages/poetry/repositories/http_repository.py:419 in calculate_sha256
     417│         )
     418│ 
   → 419│         with self._cached_or_downloaded_file(link) as filepath:
     420│             hash_name = get_highest_priority_hash_type(link.hashes, link.filename)
     421│             known_hash = None

25  /usr/lib/python3.14/contextlib.py:141 in __enter__
     139│         del self.args, self.kwds, self.func
     140│         try:
   → 141│             return next(self.gen)
     142│         except StopIteration:
     143│             raise RuntimeError("generator didn't yield") from None

24  /usr/lib/python3.14/site-packages/poetry/repositories/http_repository.py:117 in _cached_or_downloaded_file
     115│         with TemporaryDirectory(ignore_cleanup_errors=True) as temp_dir:
     116│             filepath = Path(temp_dir) / link.filename
   → 117│             self._download(
     118│                 link.url, filepath, raise_accepts_ranges=raise_accepts_ranges
     119│             )

23  /usr/lib/python3.14/site-packages/poetry/repositories/http_repository.py:102 in _download
     100│         self, url: str, dest: Path, *, raise_accepts_ranges: bool = False
     101│     ) -> None:
   → 102│         return download_file(
     103│             url,
     104│             dest,

22  /usr/lib/python3.14/site-packages/poetry/utils/helpers.py:161 in download_file
     159│             set_indicator = total_size > 1024 * 1024
     160│ 
   → 161│         for fetched_size in downloader.download_with_progress(chunk_size):
     162│             if set_indicator:
     163│                 percent = (fetched_size * 100) // total_size

21  /usr/lib/python3.14/site-packages/poetry/utils/helpers.py:238 in download_with_progress
     236│         fetched_size = 0
     237│         with atomic_open(self._dest) as f:
   → 238│             for chunk in self._iter_content_with_resume(chunk_size=chunk_size):
     239│                 if chunk:
     240│                     f.write(chunk)

20  /usr/lib/python3.14/site-packages/poetry/utils/helpers.py:216 in _iter_content_with_resume
     214│             try:
     215│                 with self._response:
   → 216│                     for chunk in self._response.iter_content(chunk_size=chunk_size):
     217│                         yield chunk
     218│                         fetched_size += len(chunk)

19  /usr/lib/python3.14/site-packages/requests/models.py:820 in generate
      818│             if hasattr(self.raw, "stream"):
      819│                 try:
   →  820│                     yield from self.raw.stream(chunk_size, decode_content=True)
      821│                 except ProtocolError as e:
      822│                     raise ChunkedEncodingError(e)

18  /usr/lib/python3.14/site-packages/urllib3/response.py:1257 in stream
     1255│                 or (self._decoder and self._decoder.has_unconsumed_tail)
     1256│             ):
   → 1257│                 data = self.read(amt=amt, decode_content=decode_content)
     1258│ 
     1259│                 if data:

17  /usr/lib/python3.14/site-packages/urllib3/response.py:1112 in read
     1110│                 return self._decoded_buffer.get(amt)
     1111│ 
   → 1112│         data = self._raw_read(amt)
     1113│ 
     1114│         flush_decoder = amt is None or (amt != 0 and not data)

16  /usr/lib/python3.14/site-packages/urllib3/response.py:1028 in _raw_read
     1026│ 
     1027│         with self._error_catcher():
   → 1028│             data = self._fp_read(amt, read1=read1) if not fp_closed else b""
     1029│             if amt is not None and amt != 0 and not data:
     1030│                 # Platform-specific: Buggy versions of Python.

15  /usr/lib/python3.14/site-packages/urllib3/response.py:1011 in _fp_read
     1009│         else:
     1010│             # StringIO doesn't like amt=None
   → 1011│             return self._fp.read(amt) if amt is not None else self._fp.read()
     1012│ 
     1013│     def _raw_read(

14  /usr/lib/python3.14/site-packages/cachecontrol/filewrapper.py:106 in read
     104│             self.__buf.write(data)
     105│         if self.__is_fp_closed():
   → 106│             self._close()
     107│ 
     108│         return data

13  /usr/lib/python3.14/site-packages/cachecontrol/filewrapper.py:86 in _close
      84│                     mmap.mmap(self.__buf.fileno(), 0, access=mmap.ACCESS_READ)
      85│                 )
   →  86│             self.__callback(result)
      87│ 
      88│         # We assign this to None here, because otherwise we can get into

12  /usr/lib/python3.14/site-packages/cachecontrol/controller.py:420 in cache_response
     418│             logger.debug(f"etag object cached for {expires_time} seconds")
     419│             logger.debug("Caching due to etag")
   → 420│             self._cache_set(cache_url, request, response, body, expires_time)
     421│ 
     422│         # Add to the cache any permanent redirects. We do this before looking

11  /usr/lib/python3.14/site-packages/cachecontrol/controller.py:320 in _cache_set
     318│             self.cache.set(
     319│                 cache_url,
   → 320│                 self.serializer.dumps(request, response, body),
     321│                 expires=expires_time,
     322│             )

10  /usr/lib/python3.14/site-packages/cachecontrol/serialize.py:60 in dumps
      58│                 data["vary"][header] = header_value
      59│ 
   →  60│         return b",".join([f"cc={self.serde_version}".encode(), self.serialize(data)])
      61│ 
      62│     def serialize(self, data: dict[str, Any]) -> bytes:

 9  /usr/lib/python3.14/site-packages/cachecontrol/serialize.py:63 in serialize
      61│ 
      62│     def serialize(self, data: dict[str, Any]) -> bytes:
   →  63│         return cast(bytes, msgpack.dumps(data, use_bin_type=True))
      64│ 
      65│     def loads(

 8  /usr/lib/python3.14/site-packages/msgpack/__init__.py:36 in packb
      34│     See :class:`Packer` for options.
      35│     """
   →  36│     return Packer(**kwargs).pack(o)
      37│ 
      38│ 

 7  msgpack/_packer.pyx:279 in msgpack._cmsgpack.Packer.pack

 6  msgpack/_packer.pyx:276 in msgpack._cmsgpack.Packer.pack

 5  msgpack/_packer.pyx:270 in msgpack._cmsgpack.Packer._pack

 4  msgpack/_packer.pyx:213 in msgpack._cmsgpack.Packer._pack_inner

 3  msgpack/_packer.pyx:270 in msgpack._cmsgpack.Packer._pack

ValueError

memoryview is too large

at msgpack/_packer.pyx:238 in msgpack._cmsgpack.Packer._pack_inner

[dosubot]

Hi @JPK64! I'm Dosu and I’m helping the poetry team.

This is a confirmed gap in Poetry's caching logic. The issue is that cachecontrol's CacheControlAdapter is mounted on all HTTP(S) requests created by Poetry's Authenticator [1], and there's no size-based bypass for large responses. When cachecontrol finishes reading the response, it tries to serialize the entire body via msgpack for its file cache — and msgpack cannot handle buffers larger than 2 GiB, which causes the ValueError: memoryview is too large.

Notably, Poetry does already bypass caching for HTTP range requests used in lazy wheel metadata extraction by sending Cache-Control: no-cache headers [2], but no equivalent logic exists for large complete file downloads.

A proper fix would likely involve one of:

1. Skipping cachecontrol caching for responses above a size threshold (e.g., by checking Content-Length before the download and using an uncached session for large files)
2. Sending Cache-Control: no-cache / no-store headers for file download requests (similar to how lazy wheel range requests already bypass caching)
3. Upstream fix in cachecontrol to gracefully handle large response bodies (e.g., by not caching bodies above the msgpack limit)

As you noted, --no-cache works because it disables the CacheControlAdapter entirely [3]. The solver.lazy-wheel setting (already enabled in your config) avoids full downloads during resolution by only fetching metadata via range requests [4], but the SHA256 calculation path still triggers a full download when the hash isn't available from the index.

This doesn't appear to have been previously reported — I didn't find any existing open issues for this specific problem.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[dimbleby]

13 /usr/lib/python3.14/site-packages/cachecontrol/filewrapper.py:86 in _close
84│ mmap.mmap(self.__buf.fileno(), 0, access=mmap.ACCESS_READ)

seems like you probably need to take this to cachecontrol

[JPK64]

I did some more digging and found out that cachecontrol seem to already have implemented a SeparateBodyFileCache for caching large files. I'll try switching poetrys FileCache to SeparateBodyFileCache locally to see if that fixes the issue. If it does, then this probably still is a poetry issue and I will provide a pull request. I'll report back later.

[JPK64]

I have drafted a pull request with a potential fix for this issue. I still need some input there.

Also, I have accidentally commented on the wrong pull request 🙃