python3Packages.setuptoolsBuildHook: deprecate 👋

[mweinelt]

The way this hook directly calls setup.py has been deprecated for years now and we need to get off of it, before it eventually breaks.

All packages should migrate to pyproject = true and put the relevant PEP517 builder (setuptools in this case) into build-system.

See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[winterqt]

@wolfgangwalther Do you think it’s a wise idea to add a check to

nixpkgs/ci/eval/default.nix

Lines 118 to 121 in f294786

	elif [[ -s "$outputDir/stderr/$myChunk" ]]; then
	echo "Nixpkgs on $system evaluated with warnings, aborting"
	kill $PPID
	fi

that allows certain kinds of warnings through? Or is there another way we should approach it?

[wolfgangwalther]

@wolfgangwalther Do you think it’s a wise idea to add a check
that allows certain kinds of warnings through?

I wouldn't go that route. That would probably become complicated - and not immediately clear to users, why some warnings are OK and others are not.

Or is there another way we should approach it?

I think you actually want CI to break here. The approach for deprecation should be:

• Remove all users of X in nixpkgs
• Deprecate X via warning for downstream users
• Eventually, remove X.

This means, that once you deprecate something, nixpkgs should already not use it anymore. Or put differently: There is not really something like "deprecation in nixpkgs".

In this case, it looks like setuptoolsBuildHook is only used in a handful of places in nixpkgs right now. So it should be rather simple to migrate the existing users, and then add the warning?

This then has the benefit that people are blocked from adding this again inside nixpkgs.

[wolfgangwalther]

In this case, it looks like setuptoolsBuildHook is only used in a handful of places in nixpkgs right now.

Scratch that, I should have grepped for format = "setuptools", I guess :D

[wolfgangwalther]

In any case, my point still stands: If the deprecation doesn't cause a CI failure, it's a pointless warning added. People will still do it in nixpkgs.

[winterqt]

Fair enough. Thank you for the detailed writeup, much appreciated.

This then has the benefit that people are blocked from adding this again inside nixpkgs.

Given we don’t have merge queues, there could be a bunch of new instances merged after, though I guess the eval check on master will quickly find those at least.

[winterqt]

I’d be willing to write a script to mechanically migrate the simple cases (probably using tree-sitter because I don’t think string replacement is good enough here, though it might just work fine 🤷‍♀️) if you want, @mweinelt.

[mweinelt]

I hate everything about this. We have no mechanism to nudge people and instead have to either develop a scripted approach or do the work in a small group that cares.

migrate the simple cases

Yeah, the problem is with the packages that don't set format at all. But I suppose with tree-sitter you can find those as well.

[natsukium]

I believe we need a deprecation notice targeted at nixpkgs developers rather than warnings for nix users. This is essentially about creating a mechanism to more actively address the "3.skill: sprintable" issue.

Currently, over 2k packages use format = "setuptools", and despite two or three years of migration efforts by our Python team and package maintainers, it's unrealistic for us to handle all these migrations alone.

While automation scripts would help, this isn't just a mechanical rewrite - it's a build system replacement that often requires manual intervention. Many packages have implicit issues that surface during migration. Ignoring dependency checks might simplify the work in some cases, but that merely postpones the underlying problems.
I previously created a simple migration script, but ended up with hundreds of unverified commits in my local repository that I couldn't validate for build.

The ideal solution would be implementing a system that alerts developers about the deprecation and encourages them to run migration scripts for packages.

[wolfgangwalther]

We have no mechanism to nudge people

Actually, we do. I think that mechanism is nixpkgs-vet and its ratchet checks. Implementing a check there could at least prevent new uses showing up.

Maybe the concept of those checks could be applied slightly wider: Currently they just confirm that (a) already confirming files continue to do so and (b) new files will confirm with the rules. We could also invent a category of checks that would fail when the related file is touched, no matter whether it failed or passed before.

This way it would be possible to force users who touch a package, e.g. for update, to also clean it up at the same time. If they don't do it, CI would fail.

Edit / posted at the same time:

I believe we need a deprecation notice targeted at nixpkgs developers rather than warnings for nix users.

Yes - that's exactly what CI is in general. So we must make CI fail for those cases. Just in a way that doesn't fail CI for everyone, just because some package is still using setuptools.

[natsukium]

The problem is that there are many packages that no one touches. I think these cannot be detected by the current nixpkgs-vet.

[mweinelt]

And the other problem is python-updates, where we touch too many packages to migrate them one by one.

[wolfgangwalther]

The problem is that there are many packages that no one touches.

OK, but that's just a different thing then. That's essentially asking for a better way to ask for help for a big treewide migration like this. But I don't see how we could enforce anything like this via any kind of CI / deprecation / warning or so.

But yeah, if not enough packages are touched at all, then the effort to implement this kind of check might not be worth it.

And the other problem is python-updates, where we touch too many packages to migrate them one by one.

This could be handled by only running this specific check on "regular" PRs, but on python-updates. Or by just flat out ignoring this CI check for python-updates and merging despite it. The nature of those kind of checks is, that this will not make other PRs fail, yet, because they don't apply for everything.

[wolfgangwalther]

While automation scripts would help, this isn't just a mechanical rewrite - it's a build system replacement that often requires manual intervention. Many packages have implicit issues that surface during migration. Ignoring dependency checks might simplify the work in some cases, but that merely postpones the underlying problems.
I previously created a simple migration script, but ended up with hundreds of unverified commits in my local repository that I couldn't validate for build.

When I deprecated substituteAll, I created some refactor tools for myself to do it at https://github.com/technowledgy/refactor-tractor. I basically did two things:

• Used ast-grep to replace substituteAll with replaceVars.
• Triggered a simplified build of the result for quick verification.

This worked very well, ultimately. I was able to deal with the biggest number of packages that way, and then had a few remaining that needed manually work.

This will probably be more complicated than that, but maybe we can start with something that tries to find leaf packages, which don't cause other rebuilds, do some automated migration, and then verify via nixpkgs-review? Would it be enough to make the packages build - or would they need to be verified manually as well?