Issue with python 3.7 EOL and miniforge

[a-sevin]

I'm trying to implement a quick procedure to deploy catkit2 on a Linux server. I wrote this script:

mkdir -p ~/miniforge
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -O ~/miniforge/miniforge.sh
bash ~/miniforge/miniforge.sh -b -u -p ~/miniforge/
rm -f ~/miniforge/miniforge.sh
source ~/miniforge/bin/activate
conda init
git clone https://github.com/spacetelescope/catkit2
cd catkit2/
conda env create --file environment.yml
conda activate catkit2

However, after running conda env create, I encounter the following error:

Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - python=3.7.16

Current channels:

  - https://conda.anaconda.org/conda-forge

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

If I modify the Python dependency version to 3.7 (which should still work for all CPython modules 3.7.X), I encounter another error:

Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package protobuf-3.20.3-py310heca2aa9_1 requires python >=3.10,<3.11.0a0, but none of the providers could be installed

Could not solve for environment specs
The following packages are incompatible
├─ protobuf ==3.20.3 * is installable with the potential options
│  ├─ protobuf 3.20.3 would require
│  │  ├─ python >=3.10,<3.11.0a0 *, which can be installed;
│  │  └─ python_abi =3.10 *_cp310, which can be installed;
│  ├─ protobuf 3.20.3 would require
│  │  ├─ python >=3.11,<3.12.0a0 *, which can be installed;
│  │  └─ python_abi =3.11 *_cp311, which can be installed;
│  ├─ protobuf 3.20.3 would require
│  │  ├─ python >=3.8,<3.9.0a0 *, which can be installed;
│  │  └─ python_abi =3.8 *_cp38, which can be installed;
│  ├─ protobuf 3.20.3 would require
│  │  └─ python_abi ==3.8 *_pypy38_pp73, which requires
│  │     └─ python =3.8 *_73_pypy, which can be installed;
│  ├─ protobuf 3.20.3 would require
│  │  ├─ python >=3.9,<3.10.0a0 *, which can be installed;
│  │  └─ python_abi =3.9 *_cp39, which can be installed;
│  └─ protobuf 3.20.3 would require
│     └─ python_abi ==3.9 *_pypy39_pp73, which requires
│        └─ python =3.9 *_73_pypy, which can be installed;
└─ python =3.7 * is not installable because there are no viable options
   ├─ python [3.7.10|3.7.12|3.7.9] conflicts with any installable versions previously reported;
   ├─ python [3.7.0|3.7.1|3.7.2|3.7.3|3.7.6] would require
   │  └─ python_abi =* *_cp37m, which conflicts with any installable versions previously reported;
   └─ python [3.7.10|3.7.12|...|3.7.9] would require
      └─ python_abi =3.7 *_cp37m, which conflicts with any installable versions previously reported.

I think the problem is that Python 3.7 has been EOL for 2 years.

FYI, the script works fine if I upgrade to Python 3.9 for example.

[ehpor]

All too aware of this. Different testbeds are manually changing the testbed version and running with that.

• HiCAT is using Python 3.7.16 since it's bound by the BMC DM and FLIR Python library support.
• SEAL is using Python 3.8.x since it's bound by KRTC dependencies which are not actively being developed anymore.
• THD2 is using Python 3.9.12. Not sure about the reasons.
• GSFC is using Python 3.8.x. Not sure about the reasons.
• I perform all my simulations on Python 3.8 / 3.10 / Python 3.11 depending on when I made the environment on that machine.

Current progress:

• BMC C++-based package is 75% done, which alleviates that bottleneck. This would make that device work on any Python version.
• FLIR camera Python libraries are available for Python 3.7, 3.8, 3.10.

Blockers for making catkit2 independent of Python version: installation PR #276. Some work needs to be done on that PR. Since that PR moves a lot of code around to different directories, it conflicts with every other PR that touches any C++ code in catkit2. It therefore requires a lot of coordination, and we haven't been able to get that yet. Also, my main focus atm is on the message broker integration, which is almost all C++-based work. Bottom line: we just need to do it.

[ehpor]

That being said, conda-forge doesn't have Python 3.7.16 on linux-x64, but the anaconda channel has. I've never used miniforge, but that's an easy way to fix that, if you really need to run Python 3.7.

[ivalaginja]

For THD2 it's the BMC DMs too, mostly.

[a-sevin]

I don't need to run Catkit2 on Python 3.7, but as it's the default, it's more of a warning that we're getting into a red zone.

I'm looking for a way to define the Python version when creating the environment without having to change the YAML file. I don't know if it's possible.

Otherwise, why include the three numbers of the Python release? The first two are enough for CPython modules compatibility.

[ivalaginja]

@a-sevin three numbers needed because of the Boston DMs. Almost every versioning decision is currently determined due to Boston software limitations. Very constraining.

[lanemeier7]

Hi, we at GSFC are using Python 3.8.20 because we use miniforge and because its not too much different from Python 3.7. Miniforge can solve for Python 3.7x but I think you need to specifically downgrade some of the dependencies like mamba to a version that is known to work. It felt easier to let it go up to 3.8 and seems like everything is working OK with some minor tweaks to importlib that are now part of the develop branch.

FYI - large institutions need to have licenses to use Anaconda since about 2020 or so.