maintainer workflow for adding wheels to existing version

[davidism]

As part of maintaining MarkupSafe, I want to upload wheels for newly supported platforms to the existing PyPI release. For example, MarkupSafe 2.0.1 is the latest version on PyPI, and I've uploaded wheels for Python 3.10 and now musllinux after the initial release as new versions of cibuildwheel support them.

I don't want to make a new release since there aren't any new features to release, and I want the builds to be against the released, tagged code.

Right now this is a very manual process for me. I check out the 2.0.1 tag as a branch and push that to make a PR against. I don't have builds enabled for pull requests, so for the PR I update the cibuildwheel version and add a on: pull_request: branches: '*' section to the workflow file. I download the build artifact and have twine upload the new files.

$ git checkout -b 2.0.1-build 2.01
$ git push --set-upstream origin 2.0.1-build

$ git checkout -b update-cibuildwheel 2.0.1-build
$ nano .github/workflows/build.yaml
$ git add .github/workflows/build.yaml
$ git commit -m 'update cibuildwheel'
$ git push --set-upstream origin update-cibuildwheel

$ twine upload -r test-pypi -s --skip-existing dist/*

Here's the latest example where I did this: pallets/markupsafe#257 I force push and change the target branch after getting the artifacts so that I merge into the normal maintenance workflow for future releases.

Do any other maintainers, cibuildwheel or otherwise, have experience with this? What is your workflow for updating cibuildwheel and building new wheels for an existing tagged release?

[joerick]

I haven't got any experience doing this, but one idea that pops into my head. Could you make a workflow_dispatch GHA workflow, with parameters for git ref (e.g. your previous version tag 2.0.1) and CIBW_BUILD...? The workflow would checkout the given ref, build, test, and upload via twine. Then you could use the Github UI to trigger the workflow for each of your old versions.

Might work..?

[davidism]

The part that's missing is needing to update the use: pypa/[email protected] against that 2.0.1 ref. workflow_dispatch looks really interesting though, but it's not clear from the docs that a dispatch variable like cibw_version could be referenced in the use line.

[henryiii]

That's what I've always done, see https://github.com/scikit-hep/boost-histogram/blob/develop/.github/workflows/wheels.yml. For musllinux, you can just build them locally, too.

You should make a branch on top of the release, and change the cibuildwheel version, then dispatch on that branch. If you have to change other things (like your TOML file to tweak skips or something), that's doable that way too. As long as you don't change the code. But you can also just use environment variables too, if you want to be extra sure the code doesn't change, but the pyproject.toml file doesn't get put in the wheel, so it should be quite safe. Your CI file is not part of your "code", so changing it is clearly safe, you could even do reproducible builds.

[henryiii]

Forgot to mention, I download then upload with twine any missing wheels from that run, rather than try to upload and use the ignore-already-uploaded option if the version is specified instead of a release tag. I think I tried that first but then decided this was safer for something I don't do all that often.

[henryiii]

Ah, yes, you could also manually set the ref to checkout, too. Missed that in @joerick's suggestion, but that would also work. It's more natural to me to work on a branch based on the original release, and GHA files (and even the pyproject.toml) are not part of the final wheel.

[joerick]

The part that's missing is needing to update the use: pypa/[email protected] against that 2.0.1 ref.

I think that in workflow_dispatch the workflow would be run in the configuration that it is at HEAD of your default branch, but you could check out old code with the checkout step.