[ci] [python-package] builds failing: "No module named build"

[jameslamb]

Description

Some time in the last day or 2, many (but not all!) Python CI jobs on macOS and Linux have started failing like this:

[INFO] found pre-compiled lib_lightgbm.so
[INFO] --- building wheel ---
/github/home/miniforge/envs/test-env/bin/python: No module named build

(example build link)

Reproducible example

This is happening on all PR CI.

I was not able to reproduce it on my M2 Mac like this:

export COMPILER=clang
export IN_UBUNTU_BASE_CONTAINER=false
export SETUP_CONDA=false
export TASK=bdist
export METHOD=wheel
export OS_NAME=macos
export PYTHON_VERSION="3.13"
export CONDA="${HOME}/miniforge3"
export PATH="${CONDA}/bin:${PATH}"
export PRODUCES_ARTIFACTS=false
export BUILD_DIRECTORY=$(pwd)
export BUILD_ARTIFACTSTAGINGDIRECTORY="${BUILD_DIRECTORY}"

rm -rf ./{build,dist}
conda env remove --yes --name test-env

./.ci/test.sh

Environment info

N/A

Additional Comments

build is managed by build-python.sh and installed with pip:

LightGBM/build-python.sh

Line 188 in 9545905

pip install --prefer-binary 'build>=0.10.0'

In CI logs, it does appear to be getting installed

Collecting build>=0.10.0
  Downloading build-1.5.0-py3-none-any.whl.metadata (5.7 kB)
Requirement already satisfied: packaging>=24.0 in /github/home/miniforge/lib/python3.13/site-packages (from build>=0.10.0) (26.2)
Collecting pyproject_hooks (from build>=0.10.0)
  Downloading pyproject_hooks-1.2.0-py3-none-any.whl.metadata (1.3 kB)
Downloading build-1.5.0-py3-none-any.whl (26 kB)
Downloading pyproject_hooks-1.2.0-py3-none-any.whl (10 kB)
Installing collected packages: pyproject_hooks, build

Successfully installed build-1.5.0 pyproject_hooks-1.2.0

I suspect that this is related to conda environment activation. The scripts are using source activate, and maybe some relevant activation variables are not exported and therefore the module can't be found inside the invoked build-python.sh?

And maybe this is only breaking now because some other dependency in the conda environment happened to be pulling in python-build before?

[jameslamb]

Looking at some recent CI logs (link), I think the issue might be that the pip being found isn't the one from the conda environment our CI jobs set up.

Requirement already satisfied: packaging>=24.0 in /Users/runner/miniforge/lib/python3.13/site-packages (from build>=0.10.0) (26.2)
...
/Users/runner/miniforge/envs/test-env/bin/python: No module named build

So build appears to be getting installed into the base environment.

Switching from pip install to python -m pip install should fix this, or at least make the issue clearer.

I think the root cause might be that... the environment no longer includes pip?????

From a successful run a few days ago:

...
partd                        1.4.2            pyhd8ed1ab_0         conda-forge
pcre2                        10.47            h13923f0_0           conda-forge
pillow                       11.3.0           py39h1fda9f2_0       conda-forge
pip                          25.2             pyh8b19718_0         conda-forge
pixman                       0.46.4           ha059160_1           conda-forge
pluggy                       1.0.0            pyhd8ed1ab_5         conda-forge       
...

(build link)

From recent failed runs:

...
partd                        1.4.2            pyhd8ed1ab_0         conda-forge
pcre2                        10.47            h13923f0_0           conda-forge
pillow                       11.3.0           py39h1fda9f2_0       conda-forge
pixman                       0.46.4           ha059160_1           conda-forge
pluggy                       1.0.0            pyhd8ed1ab_5         conda-forge
...

(build link)

[jameslamb]

Looking at CI runs on #7249, it seems clear that the root cause was "pip was no longer getting installed in the testing conda environment".

I looked around some conda-forge feedstock repos and couldn't find where this changed, but something dropped its runtime requirement on pip and LightGBM CI had apparently been depending on that 😫

[jameslamb]

Looks like PyTorch hit the same thing: pytorch/pytorch#182297