2023.02.16- MacOS - ARM64 - The environment can't be solved

[jhunkeler]

When I perform an installation on MacOS (ARM64), the environment cannot be solved.

Steps that can reproduce the error:

1. mamba env create -n stenv -f https://github.com/spacetelescope/stenv/releases/download/2023.02.16/stenv-macOS-py3.11-2023.02.16.yml
2. the following error happens:

Could not solve for environment specs
Encountered problems while solving:
 - package qt-main-5.15.8-hfe8d25c_6 requires libclang >=14.0.6,<15.0a0, but none of the providers can be installed
 - package tifffile-2023.2.3-pyhd8ed1ab_0 requires imagecodecs >=2023.1.23, but none of the providers can be installed

The environment can't be solved, aborting the operation

Originally reported by: @alechirschauer

[jhunkeler]

For the sake of completeness Conda's native solver didn't produce anything useful (does it ever?). Just several thousands lines of incomprehensible garbage followed by:

  - feature:/osx-arm64::__osx==12.6.3=0
  - feature:/osx-arm64::__unix==0=0
  - feature:|@/osx-arm64::__osx==12.6.3=0
  - feature:|@/osx-arm64::__unix==0=0
  - click=8.1.3 -> __unix
  - click=8.1.3 -> __win
  - dask-core=2023.2.0 -> click[version='>=7.0'] -> __unix
  - dask-core=2023.2.0 -> click[version='>=7.0'] -> __win
  - distributed=2023.2.0 -> click[version='>=7.0'] -> __unix
  - distributed=2023.2.0 -> click[version='>=7.0'] -> __win
  - ipykernel=6.21.2 -> __linux
  - ipykernel=6.21.2 -> __osx
  - ipykernel=6.21.2 -> __win
  - ipython=8.10.0 -> __linux
  - ipython=8.10.0 -> __osx
  - ipython=8.10.0 -> __win
  - ipywidgets=8.0.4 -> ipykernel[version='>=4.5.1'] -> __linux
  - ipywidgets=8.0.4 -> ipykernel[version='>=4.5.1'] -> __osx
  - ipywidgets=8.0.4 -> ipykernel[version='>=4.5.1'] -> __win
  - jupyter=1.0.0 -> ipykernel -> __linux
  - jupyter=1.0.0 -> ipykernel -> __osx
  - jupyter=1.0.0 -> ipykernel -> __win
  - jupyter_console=6.5.1 -> ipykernel -> __linux
  - jupyter_console=6.5.1 -> ipykernel -> __osx
  - jupyter_console=6.5.1 -> ipykernel -> __win
  - jupyter_server=2.3.0 -> terminado[version='>=0.8.3'] -> __linux
  - jupyter_server=2.3.0 -> terminado[version='>=0.8.3'] -> __osx
  - jupyter_server=2.3.0 -> terminado[version='>=0.8.3'] -> __win
  - jupyter_server_terminals=0.4.4 -> terminado[version='>=0.8.3'] -> __linux
  - jupyter_server_terminals=0.4.4 -> terminado[version='>=0.8.3'] -> __osx
  - jupyter_server_terminals=0.4.4 -> terminado[version='>=0.8.3'] -> __win
  - nbclassic=0.5.1 -> ipykernel -> __linux
  - nbclassic=0.5.1 -> ipykernel -> __osx
  - nbclassic=0.5.1 -> ipykernel -> __win
  - notebook=6.5.2 -> ipykernel -> __linux
  - notebook=6.5.2 -> ipykernel -> __osx
  - notebook=6.5.2 -> ipykernel -> __win
  - pysocks=1.7.1 -> __unix
  - pysocks=1.7.1 -> __win
  - qtconsole-base=5.4.0 -> ipykernel[version='>=4.1'] -> __linux
  - qtconsole-base=5.4.0 -> ipykernel[version='>=4.1'] -> __osx
  - qtconsole-base=5.4.0 -> ipykernel[version='>=4.1'] -> __win
  - terminado=0.17.1 -> __linux
  - terminado=0.17.1 -> __osx
  - urllib3=1.26.14 -> pysocks[version='>=1.5.6,<2.0,!=1.5.7'] -> __unix
  - urllib3=1.26.14 -> pysocks[version='>=1.5.6,<2.0,!=1.5.7'] -> __win

Your installed version is: 12.6.3

[zacharyburnett]

@kslong reportedly has a similar issue when using conda env create on an ARM64 Mac. I don't currently have an M1 machine to test with, so I will work on getting access to one.

[zacharyburnett]

@alechirschauer @jhunkeler are you able to resolve an environment with the unconstrained definition? i.e.:

mamba env create -n stenv -f https://raw.githubusercontent.com/spacetelescope/stenv/main/environment.yaml

[jhunkeler]

Yes, I was able to create an environment using environment.yml. That's good to know. Thanks!

[dvbowen]

I think I have something similar (and, apologies if it's not): I have a new Mac Studio with a "Apple M1 Ultra" chip. I installed Miniconda3-latest-MacOSX-arm64.pkg and then did as the webpage help suggested:

% conda env create --file Downloads/stenv-macOS-py3.11-2023.02.16.yml --name stenv

after which I got

Collecting package metadata (repodata.json): done
Solving environment:
Found conflicts! Looking for incompatible packages.
Examining ..... [packages]
This can take several minutes. Press CTRL-C to abort.
failed

.. and I get a VERY long list of incompatibilities. They end [to provide examples]:

Package msgpack-python conflicts for:
msgpack-python=1.0.4
distributed=2023.2.0 -> msgpack-python[version='>=1.0.0']
dask=2023.2.0 -> distributed[version='>=2023.2.0,<2023.2.1.0a0'] -> msgpack-python[version='>=1.0.0']

Package sphinxcontrib-applehelp conflicts for:
sphinx=6.1.3 -> sphinxcontrib-applehelp
sphinxcontrib-applehelp=1.0.4
sphinxcontrib-serializinghtml=1.1.5 -> sphinx -> sphinxcontrib-applehelp

Package giflib conflicts for:
giflib=5.2.1
tifffile=2023.2.3 -> imagecodecs[version='>=2023.1.23'] -> giflib[version='>=5.2.1,<5.3.0a0']
imagecodecs=2023.1.23 -> giflib[version='>=5.2.1,<5.3.0a0']

Package pycosat conflicts for:
pycosat=0.6.4
conda=22.11.1 -> pycosat[version='>=0.6.3']

Package pooch conflicts for:
pooch=1.6.0
scikit-image=0.19.3 -> scipy[version='>=1.4.1'] -> pooch

Package pypy3.9 conflicts for:
cffi=1.15.1 -> python_abi==3.9[build=*_pypy39_pp73] -> pypy3.9=7.3
cffi=1.15.1 -> pypy3.9=7.3.11

Package ruamel.yaml conflicts for:
ruamel.yaml=0.17.21
jupyter_server=2.3.0 -> jupyter_events[version='>=0.4.0'] -> ruamel.yaml
conda=22.11.1 -> ruamel.yaml[version='>=0.11.14,<0.18']

Package execnet conflicts for:
pytest-xdist=3.2.0 -> execnet[version='>=1.1']
execnet=1.9.0

Package networkx conflicts for:
networkx=3.0
scikit-image=0.19.3 -> networkx[version='>=2.2']

Package xorg-libxau conflicts for:
pillow=9.4.0 -> libxcb[version='>=1.13,<1.14.0a0'] -> xorg-libxau
xorg-libxau=1.0.9
libxcb=1.13 -> xorg-libxauThe following specifications were found to be incompatible with your system:

• feature:/osx-arm64::__osx==13.2.1=0
• feature:/osx-arm64::__unix==0=0
• feature:|@/osx-arm64::__osx==13.2.1=0
• feature:|@/osx-arm64::__unix==0=0
• click=8.1.3 -> __unix
• click=8.1.3 -> __win
• dask-core=2023.2.0 -> click[version='>=7.0'] -> __unix
• dask-core=2023.2.0 -> click[version='>=7.0'] -> __win
• distributed=2023.2.0 -> click[version='>=7.0'] -> __unix
• distributed=2023.2.0 -> click[version='>=7.0'] -> __win
• ipykernel=6.21.2 -> __linux
• ipykernel=6.21.2 -> __osx
• ipykernel=6.21.2 -> __win
• ipython=8.10.0 -> __linux
• ipython=8.10.0 -> __osx
• ipython=8.10.0 -> __win
• ipywidgets=8.0.4 -> ipykernel[version='>=4.5.1'] -> __linux
• ipywidgets=8.0.4 -> ipykernel[version='>=4.5.1'] -> __osx
• ipywidgets=8.0.4 -> ipykernel[version='>=4.5.1'] -> __win
• jupyter=1.0.0 -> ipykernel -> __linux
• jupyter=1.0.0 -> ipykernel -> __osx
• jupyter=1.0.0 -> ipykernel -> __win
• jupyter_console=6.5.1 -> ipykernel -> __linux
• jupyter_console=6.5.1 -> ipykernel -> __osx
• jupyter_console=6.5.1 -> ipykernel -> __win
• jupyter_server=2.3.0 -> terminado[version='>=0.8.3'] -> __linux
• jupyter_server=2.3.0 -> terminado[version='>=0.8.3'] -> __osx
• jupyter_server=2.3.0 -> terminado[version='>=0.8.3'] -> __win
• jupyter_server_terminals=0.4.4 -> terminado[version='>=0.8.3'] -> __linux
• jupyter_server_terminals=0.4.4 -> terminado[version='>=0.8.3'] -> __osx
• jupyter_server_terminals=0.4.4 -> terminado[version='>=0.8.3'] -> __win
• nbclassic=0.5.1 -> ipykernel -> __linux
• nbclassic=0.5.1 -> ipykernel -> __osx
• nbclassic=0.5.1 -> ipykernel -> __win
• notebook=6.5.2 -> ipykernel -> __linux
• notebook=6.5.2 -> ipykernel -> __osx
• notebook=6.5.2 -> ipykernel -> __win
• pysocks=1.7.1 -> __unix
• pysocks=1.7.1 -> __win
• qtconsole-base=5.4.0 -> ipykernel[version='>=4.1'] -> __linux
• qtconsole-base=5.4.0 -> ipykernel[version='>=4.1'] -> __osx
• qtconsole-base=5.4.0 -> ipykernel[version='>=4.1'] -> __win
• qtconsole=5.4.0 -> ipykernel[version='>=4.1'] -> __linux
• qtconsole=5.4.0 -> ipykernel[version='>=4.1'] -> __osx
• qtconsole=5.4.0 -> ipykernel[version='>=4.1'] -> __win
• terminado=0.17.1 -> __linux
• terminado=0.17.1 -> __osx
• urllib3=1.26.14 -> pysocks[version='>=1.5.6,<2.0,!=1.5.7'] -> __unix
• urllib3=1.26.14 -> pysocks[version='>=1.5.6,<2.0,!=1.5.7'] -> __win

Your installed version is: not available

[jhunkeler]

My guess is the environment files are generated on a x86_64 machine so the packages it wants to install are either (1) not available for arm64, or (2) they exist but their build history doesn't go back far enough to provide the requested versions.

Manually removing the following packages (-) from the config produces a usable environment. Everything except libgfortran=5.0.0 exists in the arm64 channel, but it isn't needed anyway because whatever package needed it linked to a different libgfortran package altogether.

The remaining packages I removed were installed as dependencies at the latest version.

--- stenv-macOS-py3.11-2023.02.16.yml   2023-02-24 16:52:46.000000000 -0500
+++ stenv-macOS-py3.11-2023.02.16.yml   2023-02-24 16:54:07.000000000 -0500
@@ -66,12 +66,10 @@
   - gst-plugins-base=1.22.0
   - gstreamer=1.22.0
   - h5py=3.8.0
-  - hdf5=1.14.0
   - heapdict=1.0.1
   - hstcal=2.7.4
   - icu=70.1
   - idna=3.4
-  - imagecodecs=2023.1.23
   - imageio=2.25.1
   - imagesize=1.4.1
   - importlib-metadata=6.0.0
@@ -106,14 +104,12 @@
   - libbrotlidec=1.0.9
   - libbrotlienc=1.0.9
   - libcblas=3.9.0
-  - libclang=13.0.1
   - libcurl=7.88.0
   - libcxx=14.0.6
   - libdeflate=1.17
   - libedit=3.1.20191231
   - libev=4.33
   - libffi=3.4.2
-  - libgfortran=5.0.0
   - libgfortran5=11.3.0
   - libglib=2.74.1
   - libiconv=1.17
@@ -234,7 +230,6 @@
   - stack_data=0.6.2
   - tblib=1.7.0
   - terminado=0.17.1
-  - tifffile=2023.2.3
   - tinycss2=1.2.1
   - tk=8.6.12
   - toml=0.10.2

[dvbowen]

Just got into the office to try out this solution. Looks like it works fine. Many thanks!

[mperrin]

Hi, I just wanted to check if there is any news or timeline update on having a stenv build that's compatible with Apple Silicon systems (M1 etc).

Is the recommended path currently still to install from the base environment.yml file? Thanks.

[zacharyburnett]

Hi, I just wanted to check if there is any news or timeline update on having a stenv build that's compatible with Apple Silicon systems (M1 etc).

Is the recommended path currently still to install from the base environment.yml file? Thanks.

Hello! Unfortunately, for the meantime that method is recommended. I'm currently working on manually building files for M1, and we're in the process of setting up ARM64 runners to automate the process as well. Hopefully those can be ready soon!

[mperrin]

Super, thanks for the info. And the near-instant response :-). I'll go with that method for now, and will stay tuned for future improvements.

(I've got a Mac Studio with M1 Max at home, and it's impressively fast...)

[mperrin]

Hi and happy new year, I'm checking again whether there may be any update on this. Is the recommended stenv installation method on ARM Macs still to use the base environment.yml? Any update on timelines for having ARM builds along with the regular X64 Intel ones? Cheers and thanks.

[zacharyburnett]

Happy new year! Yes, using environment.yml is still the recommended way on ARM64; we have implemented ARM64 runners locally, but have yet to roll them out to stenv. Unfortunately, I don't have an estimate on when that might be, but I'll be sure to let you know!