[ci] adapt to scikit-learn ClassifierChain changes, fix {fs} install, work around pyarrow type-checking issues

[jameslamb]

Another day, more broken CI 😅

Fixes #7238

Fixes #7235

Fixes #7208 (again)

See those issues for details.

[jameslamb]

mypy is now failing like this:

mypy.....................................................................Failed
- hook id: mypy
- duration: 6.4s
- exit code: 1

python-package/lightgbm/basic.py:3118: error: Module has no attribute "all"  [attr-defined]
python-package/lightgbm/basic.py:3118: error: Module has no attribute "equal"  [attr-defined]
Found 2 errors in 1 file (checked 9 source files)

(build link)

Probably related to pyarrow 24.0 being pushed a few hours ago: https://pypi.org/project/pyarrow/24.0.0/

I won't be able to return to this until maybe 12 hours from now, so if other maintainers look before I do, please feel free to just push directly to my PR here. Otherwise, I'll try to fix this later.

---

update (Apr 21): documented in #7238, workaround pushed here**

[jameslamb]

Ok this is finally passing! I think it will resolve all of our current CI-blocking issues.

[borchero]

Thanks for the fix!

[borchero]

Just an fyi because of the TODO comment: I'm still planning on unifying our data frame intake processing via Narwhals at some point; this would alleviate the need for this.

[jameslamb]

Thanks for the comment.

I'm a bit skeptical of a narwhals dependency in lightgbm. We have so little data frame manipulation here (mostly just read-only access to types, size, and the underlying data), I'm unsure that narwhals will bring enough simplification or other benefits to justify adding a dependency and all the burden that comes with that (including effectively losing control of our minimum pandas and pyarrow versions).

But I'm willing to be convinced! And anyway, we can continue that conversation on a draft PR or #6204