[REWRITE] cookiecutter + cruft template with modern tooling

[zacharyburnett]

building on suggestions from @sosey and @pllim and @braingram, this is a total rewrite of this repository to turn it into a Cookiecutter template with best practices, that we can use with cruft to keep downstream templated repositories up-to-date.

Feedback is welcome! I split these changes into several commits; see https://github.com/zacharyburnett/stsci-package-template/tree/modernize for a current state of the template in this PR

templated example of a pure-Python package (stpipe) here: spacetelescope/stpipe#306

and a Python package with C extensions (stsci.imagestats): spacetelescope/stsci.imagestats#93

Tasks

• choice of automated docs publishing:
  • ReadTheDocs
  • GitHub Pages
• option to manage changelog with towncrier
  • include automated GitHub Actions workflow to validate PRs
• choice of task runner:
  • nox
  • tox
• choice of language:
  • Python
    • automated workflows in GitHub Actions
      • build
      • test
    • choice of build backend:
      • hatchling
      • setuptools
        • option to use C extensions
  • C
  • Rust
    • option to set up Python bindings with pyo3 and maturin
  • none
• include a default Dependabot config
• option to include automated pull request labeling with GitHub Actions workflow
• include a default .gitignore
• include a default pull request template
• integrate cruft into template - see example at https://github.com/Roman-Supernova-PIT/snappl/blob/main/.github/workflows/sub_package_update.yml

Notes

• centralize workflows at https://github.com/spacetelescope/stsci-public-actions?

[sosey]

add our default to LICENSE at the top: "BSD 3-Clause License"
Copyright (c) {% now 'utc', '%Y' %}

[sosey]

@zacharyburnett you want me to pr some of the standard file/templates I already have to your branch? Or elsewhere?

[zacharyburnett]

@zacharyburnett you want me to pr some of the standard file/templates I already have to your branch? Or elsewhere?

Sure, feel free to PR against this branch! Thanks!

[zacharyburnett]

I removed the built docs, because as far as I understand this repo does not need a dedicated RTD page

[zacharyburnett]

as far as the pre-commit hooks, the base .pre-commit-config.yaml in the root is for the template itself; the one that's copied to the templates is templates/.pre-commit-config.yaml

[zacharyburnett]

If I pip install -e .[test] the package doesn't install pytest since the test dependencies are in a group. Do any packages we maintain use dependency groups? Why switch to these?

I think dependency-groups are the direction we should be moving in instead of extras / optional-dependencies, as the way we're using test and docs aren't really features of the package but instead dev environments. Extras are installable by users, but IMO test and docs should only be accessible from a local clone of the source. Also, installing an extra requires installing the package itself (pip install .[test]), which makes installing reproducible frozen builds more complicated; installing a dependency group does not require installing the package, only that the command be able to find the pyproject.toml file (pip install --group test). So, IMO something like sdp SHOULD be an extra since those deps are used in the actual shipped code (scripts and the like), but test and docs should be groups because they're for devs

However, you do make a good point that for initial adoption and to reduce friction with maintainers, we can implement the first rollout of this template with extras, and then later change them to dependency groups and have cruft pick those up as PRs

[braingram]

as far as the pre-commit hooks, the base .pre-commit-config.yaml in the root is for the template itself; the one that's copied to the templates is templates/.pre-commit-config.yaml

Thanks. This was in reference to the rendered .pre-commit. For the test pure python project it generated:

ci:
  autofix_prs: false
  autoupdate_schedule: monthly

exclude: ".*\\.asdf$"

repos:
  # basic checks
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v6.0.0
    hooks:
      - id: check-added-large-files
      - id: check-case-conflict
      - id: check-merge-conflict
      - id: check-symlinks
      - id: detect-private-key
      - id: end-of-file-fixer
      - id: trailing-whitespace
  # validate change log and fragments in `changes/`
  - repo: https://github.com/twisted/towncrier
    rev: 25.8.0
    hooks:
      - id: towncrier-check
  # spell-checking
  - repo: https://github.com/codespell-project/codespell
    rev: v2.4.1
    hooks:
      - id: codespell
        args:
          - --write-changes
          - --summary
        additional_dependencies:
          - tomli
  # lint GitHub Actions workflows
  - repo: https://github.com/rhysd/actionlint
    rev: v1.7.11
    hooks:
      - id: actionlint

Did it render the wrong template?

[zacharyburnett]

Did it render the wrong template?

yes it looks like the symlinks were pointing to the wrong file 😅

[zacharyburnett]

we can use some combination of taplo, tombi, validate-pyproject, and / or tomllint here

[braingram]

Thanks for making the updates! Testing the template by re-making the package shows most of the issues are resolved. There are some remaining/new issues:

• .gitignore: thanks for cleaning this up, it still contains many lines for unused tools
• docs fail to build with error "sphinx.errors.ExtensionError: Could not import extension numfig (exception: No module named 'numfig')"
• precommit fails on fresh package and reformats a number of files. Even after reformatting there are errors that prevent commits.

[zacharyburnett]

there's also the lingering question of whether the "recommendation" of this template should be to do type checking with mypy in the pre-commit config, since a lot of our packages don't do type hinting; perhaps that should be put into a task runner like just or nox to be run on demand, instead of being a requirement for code merging

@sydduckworth has been doing some work with type checking with Pyrefly and establishing baselines for packages adopting type checking; maybe we can explore that in a future PR that adds a new option to the cookiecutter

[sosey]

there's also the lingering question of whether the "recommendation" of this template should be to do type checking with mypy in the pre-commit config; perhaps that could be something put into a task runner like just or nox. @sydduckworth has been doing some work with type checking with Pyrefly and establishing baselines for packages adopting type checking; maybe we can explore that in a future PR that adds a new option to the cookiecutter

I think we should consider this in a future PR, type checking isn't a standard across our packages