Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sketch new "Maintenance tasks and scripts" contributing dev docs #21061

Merged
merged 3 commits into from
Jun 18, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
title: Maintenance tasks and scripts
sidebar_position: 6
---

There's a variety of maintenance tasks that happen on different frequencies.

## Update the default/known versions of a built-in tool/subsystem

### External tools (downloaded executables)

Some tools use [the `ExternalTool` class](../../writing-plugins/the-rules-api/installing-tools#externaltool-install-pre-compiled-binaries) to download a binary from the internet, verify its size and hash, and then execute it.

To update these:

1. For each platform and version to add:
1. Download the archive/binary.
2. Verify it: check signatures and/or hashes if available.
3. Compute the sha256 hash and byte length. For example, if it's called `archive.zip`: `tee >(shasum -a 256) >(wc -c) > /dev/null < archive.zip`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the one line approach with curl, such as:

You can easily get the len and checksum with the following command:
curl -L $URL | tee >(wc -c) >(shasum -a 256) >/dev/null

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered that style, but I thought step 2 was fairly important: if Pants distributed an obviously-incorrect artifact by default (e.g. someone has uploaded a different binary that doesn't match signature), that would be very bad behaviour on our part!

It seems more acceptable/less egregious if a user does it to themselves, in their own local config.

2. Apply the new values:
1. Adjust `default_version` to the new version.
2. Add or replace to `default_known_versions` using the hashes and lengths above (for some tools we don't preserve older versions, especially if they have strong backwards compatibility guarantees, while for others we do retain older versions).

Example: [#20469](https://github.com/pantsbuild/pants/pull/20469).

#### PEX

The PEX external tool is a bit special, because it also appears as a requirement of the Pants project itself in `3rdparty/python/requirements.txt`. To update pex, do both:

1. Update the `pex-cli` subsystem, as above (in `src/python/pants/backend/python/util_rules/pex_cli.py`).
2. Update the requirements file and run `pants generate-lockfiles --resolve=python-default` to update Pants' own lockfile.

Example: [#20782](https://github.com/pantsbuild/pants/pull/20782).

#### Terraform

The `build-support/bin/terraform_tool_versions.py` script can help update Terraform versions.

### Python tools

Some tools use `PythonToolBase` to install executable PyPI packages, using a lockfile. Pants packages a default lockfile with a specific version of the package. To update these:

1. Adjust `default_requirements` and/or `default_interpreter_constraints` as required.
2. Run `build-support/bin/generate_builtin_lockfiles.py $scope`, where `$scope` is the `options_scope` of the subsystem class.

Example: [#20924](https://github.com/pantsbuild/pants/pull/20924).

### JVM tools

Some tools use `JVMToolBase` to install executable JVM packages, using a lockfile. Pants packages a default lockfile with a specific version of the package. To update these:

1. Adjust `default_version` and/or `default_artifacts` as required.
2. Run `build-support/bin/generate_builtin_lockfiles.py $scope`, where `$scope` is the `options_scope` of the subsystem class.

Example: none yet.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find a single JVMToolBase that we've updated since it was first added, i.e. no pull request that does the update. I might've missed one though. Let me know if you know of one.


### JS tools

Some tools use `NodeJSToolBase` to install executable npm packages. To update these:

1. Update `default_version`. That's all.

Example: [#21007](https://github.com/pantsbuild/pants/pull/21007).

## Generate a new JSON schema file

Some editors can use JSON Schema for better completions (etc.) when editing TOML files like `pants.toml`. To generate such a schema:

1. Run `./pants help-all > all-help.json`
2. Run `./pants run build-support/bin/generate_json_schema.py -- --all-help-file=all-help.json`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the ./ prefix here intentional? The previous pants invocation was without.. I guess it'll be a matter of preference to whom ever runs the command, I just wonder if we should be consistent/prefer one over the other in the documentation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, just copied from the generate_json_schema.py script. Will adjust. Good catch.

3. For a new release, upload the resulting file to https://www.schemastore.org/json/

## Update or create FaaS complete platforms files

The function-as-a-service (FaaS) subsystems provide some built-in PEX complete platforms JSON files, for specific runtimes. To update or create these:

### AWS Lambda

1. Adjust `PythonAwsLambdaRuntime.known_runtimes` as required
2. Run `build-support/bin/generate_faas_complete_platforms.py` to create any new files and update the existing ones, using AWS's published docker images

Example: [#21004](https://github.com/pantsbuild/pants/pull/21004).
Comment on lines +77 to +82
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the section that (hopefully!) satisfies the original request from #21004 (review)


### Google Cloud Functions

Complete platforms are not provided for GCF yet: [#18195](https://github.com/pantsbuild/pants/issues/18195).
Loading