-
-
Notifications
You must be signed in to change notification settings - Fork 645
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sketch new "Maintenance tasks and scripts" contributing dev docs #21061
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
--- | ||
title: Maintenance tasks and scripts | ||
sidebar_position: 6 | ||
--- | ||
|
||
There's a variety of maintenance tasks that happen on different frequencies. | ||
|
||
## Update the default/known versions of a built-in tool/subsystem | ||
|
||
### External tools (downloaded executables) | ||
|
||
Some tools use [the `ExternalTool` class](../../writing-plugins/the-rules-api/installing-tools#externaltool-install-pre-compiled-binaries) to download a binary from the internet, verify its size and hash, and then execute it. | ||
|
||
To update these: | ||
|
||
1. For each platform and version to add: | ||
1. Download the archive/binary. | ||
2. Verify it: check signatures and/or hashes if available. | ||
3. Compute the sha256 hash and byte length. For example, if it's called `archive.zip`: `tee >(shasum -a 256) >(wc -c) > /dev/null < archive.zip`. | ||
2. Apply the new values: | ||
1. Adjust `default_version` to the new version. | ||
2. Add or replace to `default_known_versions` using the hashes and lengths above (for some tools we don't preserve older versions, especially if they have strong backwards compatibility guarantees, while for others we do retain older versions). | ||
|
||
Example: [#20469](https://github.com/pantsbuild/pants/pull/20469). | ||
|
||
#### PEX | ||
|
||
The PEX external tool is a bit special, because it also appears as a requirement of the Pants project itself in `3rdparty/python/requirements.txt`. To update pex, do both: | ||
|
||
1. Update the `pex-cli` subsystem, as above (in `src/python/pants/backend/python/util_rules/pex_cli.py`). | ||
2. Update the requirements file and run `pants generate-lockfiles --resolve=python-default` to update Pants' own lockfile. | ||
|
||
Example: [#20782](https://github.com/pantsbuild/pants/pull/20782). | ||
|
||
#### Terraform | ||
|
||
The `build-support/bin/terraform_tool_versions.py` script can help update Terraform versions. | ||
|
||
### Python tools | ||
|
||
Some tools use `PythonToolBase` to install executable PyPI packages, using a lockfile. Pants packages a default lockfile with a specific version of the package. To update these: | ||
|
||
1. Adjust `default_requirements` and/or `default_interpreter_constraints` as required. | ||
2. Run `build-support/bin/generate_builtin_lockfiles.py $scope`, where `$scope` is the `options_scope` of the subsystem class. | ||
|
||
Example: [#20924](https://github.com/pantsbuild/pants/pull/20924). | ||
|
||
### JVM tools | ||
|
||
Some tools use `JVMToolBase` to install executable JVM packages, using a lockfile. Pants packages a default lockfile with a specific version of the package. To update these: | ||
|
||
1. Adjust `default_version` and/or `default_artifacts` as required. | ||
2. Run `build-support/bin/generate_builtin_lockfiles.py $scope`, where `$scope` is the `options_scope` of the subsystem class. | ||
|
||
Example: none yet. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I couldn't find a single |
||
|
||
### JS tools | ||
|
||
Some tools use `NodeJSToolBase` to install executable npm packages. To update these: | ||
|
||
1. Update `default_version`. That's all. | ||
|
||
Example: [#21007](https://github.com/pantsbuild/pants/pull/21007). | ||
|
||
## Generate a new JSON schema file | ||
|
||
Some editors can use JSON Schema for better completions (etc.) when editing TOML files like `pants.toml`. To generate such a schema: | ||
|
||
1. Run `./pants help-all > all-help.json` | ||
2. Run `./pants run build-support/bin/generate_json_schema.py -- --all-help-file=all-help.json` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah, just copied from the |
||
3. For a new release, upload the resulting file to https://www.schemastore.org/json/ | ||
|
||
## Update or create FaaS complete platforms files | ||
|
||
The function-as-a-service (FaaS) subsystems provide some built-in PEX complete platforms JSON files, for specific runtimes. To update or create these: | ||
|
||
### AWS Lambda | ||
|
||
1. Adjust `PythonAwsLambdaRuntime.known_runtimes` as required | ||
2. Run `build-support/bin/generate_faas_complete_platforms.py` to create any new files and update the existing ones, using AWS's published docker images | ||
|
||
Example: [#21004](https://github.com/pantsbuild/pants/pull/21004). | ||
Comment on lines
+77
to
+82
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the section that (hopefully!) satisfies the original request from #21004 (review) |
||
|
||
### Google Cloud Functions | ||
|
||
Complete platforms are not provided for GCF yet: [#18195](https://github.com/pantsbuild/pants/issues/18195). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really like the one line approach with
curl
, such as:pants/src/python/pants/core/target_types.py
Lines 225 to 228 in e44697e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I considered that style, but I thought step 2 was fairly important: if Pants distributed an obviously-incorrect artifact by default (e.g. someone has uploaded a different binary that doesn't match signature), that would be very bad behaviour on our part!
It seems more acceptable/less egregious if a user does it to themselves, in their own local config.