Skip to content

Commit

Permalink
Sketch new "Maintenance tasks and scripts" contributing dev docs (#21061
Browse files Browse the repository at this point in the history
)

Fixes #21034 

This adds a new page to the "contributing > development" section of the
docs, that describes various common tasks that happen as part of the
development flow. This is partly reminding us of various scripts in
`build-support/bin/`, but not entirely.

This is just a starting point: it doesn't list all the scripts in
`build-support/bin/` (although some aren't intended for human use
anyway, I believe), probably doesn't list all the tasks. Easy to expand
in future!

Rendered table of contents: 


![image](https://github.com/pantsbuild/pants/assets/1203825/99b5b0b4-a820-4a47-95ac-473270e7471c)
  • Loading branch information
huonw authored Jun 18, 2024
1 parent 7970d30 commit 6f3a311
Show file tree
Hide file tree
Showing 2 changed files with 119 additions and 2 deletions.
4 changes: 2 additions & 2 deletions build-support/bin/generate_json_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
Live run:
$ ./pants help-all > all-help.json
$ ./pants run build-support/bin/generate_json_schema.py -- --all-help-file=all-help.json
$ pants help-all > all-help.json
$ pants run build-support/bin/generate_json_schema.py -- --all-help-file=all-help.json
"""
import argparse
import itertools
Expand Down
117 changes: 117 additions & 0 deletions docs/docs/contributions/development/maintenance-tasks-and-scripts.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
---
title: Maintenance tasks and scripts
sidebar_position: 6
---

There's a variety of maintenance tasks that happen on different frequencies.

## Update the default/known versions of a built-in tool/subsystem

### External tools (downloaded executables)

Some tools use [the `ExternalTool` class](../../writing-plugins/the-rules-api/installing-tools#externaltool-install-pre-compiled-binaries) to download a binary from the internet, verify its size and hash, and then execute it.

To update these:

1. For each platform and version to add:
1. Download the archive/binary.
2. Verify it: check signatures and/or hashes if available.
3. Compute the sha256 hash and byte length. For example, if it's called `archive.zip`: `tee >(shasum -a 256) >(wc -c) > /dev/null < archive.zip`.
2. Apply the new values:
1. Adjust `default_version` to the new version.
2. Add or replace to `default_known_versions` using the hashes and lengths above (for some tools we don't preserve older versions, especially if they have strong backwards compatibility guarantees, while for others we do retain older versions).

Example: [#20469](https://github.com/pantsbuild/pants/pull/20469).

#### PEX

The PEX external tool is a bit special, because it also appears as a requirement of the Pants project itself in `3rdparty/python/requirements.txt`. To update pex, do both:

1. Update the `pex-cli` subsystem, as above (in `src/python/pants/backend/python/util_rules/pex_cli.py`).
2. Update the requirements file and run `pants generate-lockfiles --resolve=python-default` to update Pants' own lockfile.

Example: [#20782](https://github.com/pantsbuild/pants/pull/20782).

#### Terraform

The `build-support/bin/terraform_tool_versions.py` script can help update Terraform versions.

### Python tools

Some tools use `PythonToolBase` to install executable PyPI packages, using a lockfile. Pants packages a default lockfile with a specific version of the package. To update these:

1. Adjust `default_requirements` and/or `default_interpreter_constraints` as required.
2. Run `build-support/bin/generate_builtin_lockfiles.py $scope`, where `$scope` is the `options_scope` of the subsystem class.

Example: [#20924](https://github.com/pantsbuild/pants/pull/20924).

### JVM tools

Some tools use `JVMToolBase` to install executable JVM packages, using a lockfile. Pants packages a default lockfile with a specific version of the package. To update these:

1. Adjust `default_version` and/or `default_artifacts` as required.
2. Run `build-support/bin/generate_builtin_lockfiles.py $scope`, where `$scope` is the `options_scope` of the subsystem class.

Example: none yet.

### JS tools

Some tools use `NodeJSToolBase` to install executable npm packages. To update these:

1. Update `default_version`. That's all.

Example: [#21007](https://github.com/pantsbuild/pants/pull/21007).

## Generate a new JSON schema file

Some editors can use JSON Schema for better completions (etc.) when editing TOML files like `pants.toml`. To generate such a schema:

1. Run `pants help-all > all-help.json`
2. Run `pants run build-support/bin/generate_json_schema.py -- --all-help-file=all-help.json`
3. For a new release, upload the resulting file to https://www.schemastore.org/json/

## Update or create FaaS complete platforms files

The function-as-a-service (FaaS) subsystems provide some built-in PEX complete platforms JSON files, for specific runtimes. To update or create these:

### AWS Lambda

1. Adjust `PythonAwsLambdaRuntime.known_runtimes` as required
2. Run `build-support/bin/generate_faas_complete_platforms.py` to create any new files and update the existing ones, using AWS's published docker images

Example: [#21004](https://github.com/pantsbuild/pants/pull/21004).

### Google Cloud Functions

Complete platforms are not provided for GCF yet: [#18195](https://github.com/pantsbuild/pants/issues/18195).

## Cherry-pick a pull request to an older version

We maintain multiple versions, with `main` being our development branch, and various `2.*.x` branches for the stable versions (see [Release strategy](../releases/release-strategy.mdx) for more details).

### Cherry-picking a new pull request

When a change needs to land in `main` but also one or more older versions, the usual process is:

1. Create or review the pull request against `main` as usual
2. Label it as `needs-cherrypick` and set milestone to the oldest release to which it should be cherry-picked
3. Merge the pull request as normal
4. At this point, automation kicks in and attempts to cherrypick the merged commit to the release in the milestone and any newer ones.
5. The automation opens pull requests targeting each of the relevant `2.*.x` branches for which cherry-picking succeeds.
6. If the automation fails to do a cherry-pick, it will mark the PR as `auto-cherry-picking-failed`
7. In either case, the automation will add a comment describing what happened to the original pull request.

For example, suppose `main` is for `2.23.x` and we're still maintaining `2.20.x`, `2.21.x` and `2.22.x`. If a pull request is labelled `needs-cherrypick` and has milestone `2.21.x`, then merging it will attempt to cherry pick to `2.21.x` and `2.22.x`.

The process may fail in one of two ways:

- The cherry-picking process failed, and tagged the PR with `auto-cherry-picking-failed`: follow the instructions in the comment on the pull request. (This likely means there are merge conflicts that require manual resolution.)
- the cherry-pick hasn't (yet) run: trigger the automation manually by going to [the GitHub Action](https://github.com/pantsbuild/pants/actions/workflows/auto-cherry-picker.yaml), clicking on the "Run workflow" button, and providing the PR number.

### Cherry-picking a merged pull request

A pull request might merged without being configured for cherry-picking, and we decide later that it should be. To cherry-pick in this case:

1. Label the pull request as `needs-cherrypick` and set milestone to the oldest release to which it should be cherry-picked
2. Trigger the automation manually by going to [the GitHub Action](https://github.com/pantsbuild/pants/actions/workflows/auto-cherry-picker.yaml), clicking on the "Run workflow" button, and providing the PR number.
3. As above, the automation may (partially) succeed or fail, and will leave a comment describing what happened.

0 comments on commit 6f3a311

Please sign in to comment.