Verifying that the release process works for all maintainers

[twpayne]

Creating a release should require only creating a tag and pushing it to GitHub, as described in the developer documentation. However, so far I'm the only person to have done this, so it's possible that the release process is broken for other maintainers.

@bradenhilton would you be willing to cut the next release of chezmoi? This would be a version bump to v.2.26.0 as we've added several features since v2.25.0 (as well as fixing a few bugs). The target release date would be sometime around the weekend of 29/30 October, but flexible.

For the first time doing this, it's best that we're online at the same time, so we can fix any problems quickly.

You can run basic tests with make test-release on a Linux machine, but creating an actual release triggers a number of events in systems beyond our control that we cannot test for in advance, for example:

• It creates a new GtiHub release, which is publicly visible.
• Homebrew and Chocolately automation spots the new tag in a couple of hours and creates new packages.
• The release process relies on a number of third-party services working correctly at the time of release. For example, the Go module proxy, the DNS servers for vanity module imports (I'm looking at you, filippo.io, you've failed at least twice), GitHub Actions, and the various package management systems used in chezmoi's platform testing.

tl;dr if the release process works, it's effortless. If the release process fails then you have to make a fast decision whether to delete the release and try again (hoping that you caught the problem before it escaped to the outside world) or polluting the release history with a broken release by creating a new patch release.

Would you be available around the 29/30 October to verify and make the next release of chezmoi? No worries if not available for any reason and apologies for asking in public.

[bradenhilton]

I'm still not too familiar with GitHub releases but I'm happy to give it a go.

Some questions if you don't mind:

• What are some of the telltale signs of failure to look out for? Is it just a case of pushing the tag and observing the status in actions?
• Is deleting the release as simple as deleting it from GitHub and running git push --delete origin v2.26.0?

Also, if you'd prefer, you can release 2.26.0, and I can release 2.26.1 after that. If something goes wrong we can either delete 2.26.1 and try again or release 2.26.2.

Either is fine with me.

[twpayne]

Thanks for considering this :)

• What are some of the telltale signs of failure to look out for? Is it just a case of pushing the tag and observing the status in actions?

Yes, although you don't even need to observe the status: GitHub will notify you if the process fails. It's just that if the process does fail then you might need to take some action. You need to be near your keyboard for about an hour after pushing the tag.

Here's an example run: https://github.com/twpayne/chezmoi/actions/runs/3242767562

This run passed, but to give a bit of background on the steps:

• The test-release job builds everything, and just skips the signing and publishing steps. This catches most problems early.
• Only if the test-release job passes does the release step run. This builds, signs, and then publishes, so problems with the signing and publishing are only discovered here.

The worst case is that the release job fails partially through the publishing steps. You then have to clean up whatever has been published. Luckily, I don't recall this happening for a long time. Normally, as you say, deleting the tag from GitHub is sufficient. You can then push commits to fix the problem and re-tag the newer version. If the GitHub release gets published then you might have to delete that via the GitHub web interface, but the release publish is one of the very last steps of the process, so there's not much that can fail after it has run.

Also, if you'd prefer, you can release 2.26.0, and I can release 2.26.1 after that. If something goes wrong we can either delete 2.26.1 and try again or release 2.26.2.

This might make sense. I've recently made a number of changes to the release process (recently adding signing with cosign, and I'm wondering if I can improve the automatic generation of release notes in the next week or so). Let's re-evaluate next week.

[twpayne]

OK, for info I'll release 2.26.0 now. We might need to release 2.26.1 on Tuesday 1 November as there's a Go security fix coming.

[bradenhilton]

Release in progress :)

[twpayne]

Yay! It looks like the process worked:

https://github.com/twpayne/chezmoi/releases/tag/v2.27.0

https://github.com/twpayne/chezmoi/actions/runs/3427774351

Unfortunately as the tag doesn't have author information, it looks like GitHub assumes the tag was pushed by the repo owner. Let's see if we can get the release properly credited next time.

[bradenhilton]

Doesn't that require annotated tags? Unless there's something on the GitHub end to configure.

[twpayne]

I don't know. Another possibility is that as goreleaser creates the GitHub Release using a GitHub Personal Access Token associated with my account. GitHub's create release REST API does not include an author field. Investigation required.

[bradenhilton]

Looking at the workflow page, it knows that I pushed the tag. but the tag is on a commit that you authored, so that might be a factor also.

[twpayne]

@halostatue we should probably release a new version of chezmoi early next week. We've accumulated sufficient changes to justify a v2.29.0 tag.

Would you be willing to do the release of v2.29.0? I would suggest creating and pushing the new tag around Monday-ish at a time convenient to you.

[halostatue]

FYI: I’ve tried a few different things and I am currently hitting an error with make test-release. I hit this in both my attempt to make a Dockerfile and in a Debian 11 VM. I can probably install an Ubuntu 20.04 VM, but it will take me more time.

  ⨯ release failed after 2m50s               error=failed to build for linux_amd64_v1: exit status 1: # runtime/cgo
gcc: error: unrecognized command-line option '-m64'

I suspect that this is either something new to gcc 10.2 or because I’m running on arm64 (M1 Pro).

[halostatue]

If we want to release a 2.30.0 tag in the next couple of days, I would suggest that you or @bradenhilton trigger it while I figure out what's going on with make test-release.

[twpayne]

It's OK to just push the v2.30.0 tag. The test-release job runs as part of every CI triggered by a code change.

Running test-release locally is likely to only work on Ubuntu distributions.

[halostatue]

I pushed the tag v2.30.0.

[twpayne]

Thank you! The action is running at https://github.com/twpayne/chezmoi/actions/runs/4107699348. It's really slow, sorry, and takes about 45-50 minutes to complete.

Basically, it's cross-compiling binaries for several tens of architectures on very slow GitHub Action runners. Speeding it up somehow is on my to-do list.