Merge branch 'main' into port_cli_aliasing_to_rust

build-support/flake8/.flake8

@@ -19,7 +19,7 @@ extend-ignore:
   # Unnecessary dict call - rewrite as a literal
   C408,
   # Temporarily exclude during Python upgrade
-  C420,E721
+  E721
 
 [flake8:local-plugins]
 extension =

docs/docs/getting-started/installing-pants.mdx

@@ -35,16 +35,14 @@ bin i github.com/pantsbuild/scie-pants ~/.local/bin/pants
 
 If you have difficulty installing Pants, see our [getting help](/community/getting-help) for community resources to help you resolve your issue.
 
-:::tip Upgrading Pants
+## Upgrading Pants
+
 The `pants` launcher binary will automatically install and use the Pants version specified in `pants.toml`, so upgrading Pants in a repo is as simple as editing `pants_version` in that file.
 
-To upgrade the `pants` launcher binary itself, run
+To upgrade the `pants` launcher binary itself, either:
 
-```
-SCIE_BOOT=update pants
-```
-
-:::
+- Use the package manager you used to install Pants. For example, with Homebrew: `brew update && brew upgrade pantsbuild/tap/pants`.
+- Use its built-in self-update functionality: `SCIE_BOOT=update pants`.
 
 ## Running Pants from sources
 

docs/notes/2.22.x.md

@@ -135,6 +135,10 @@ The default version of `semgrep` used by the `pants.backends.experimental.tool.s
 
 Setting [the `orphan_files_behaviour = "ignore"` option](https://www.pantsbuild.org/2.22/reference/subsystems/yamllint#orphan_files_behavior) the `pants.backend.experimental.tools.yamllint` backend is now properly silent. It previously showed spurious warnings.
 
+#### Docker
+
+Pants now correctly extracts the image ID when passing the equivalent of `--provenance=false` to `docker buildx`.
+
 ### Plugin API changes
 
 The `PythonToolRequirementsBase` and `PythonToolBase` classes now have a new `help_short` field. Subclasses should now use `help_short` instead of the `help` field. The `help` field will be automatically generated using `help_short`, and will include the tool's default package version and provide instructions on how to override this version using a custom lockfile.

docs/notes/2.24.x.md

@@ -2,25 +2,26 @@
 
 Pants 2 is a fast, scalable, user-friendly build system for codebases of all sizes. It's currently focused on Python, Go, Java, Scala, Kotlin, Shell, and Docker, with support for other languages and frameworks coming soon.
 
-Individuals and companies can now [sponsor Pants financially](https://www.pantsbuild.org/sponsorship).
-
 Pants is an open-source project that is not owned or controlled by any one company or organization, and does incur some expenses. These expenses are managed by Pants Build, a non-profit that was established for this purpose. This non-profit's only source of revenue is sponsorship by individuals and companies that use Pants.
 
 We offer [formal sponsorship tiers for companies](https://www.pantsbuild.org/sponsorship), as well as individual sponsorships via [GitHub](https://github.com/sponsors/pantsbuild).
 
+Thank you to [Klayvio](https://www.klaviyo.com/) and [Normal Computing](https://normalcomputing.ai/) for their Platinum tier support through throughout this release.
+
 ## What's New
 
 ### Highlights
 
 - Support for Python 3.13
 - `pants export --bin` allows exporting more tools for use outside Pants
+- A new experimental Python Provider backend using [Python Build Standalone](https://gregoryszorc.com/docs/python-build-standalone/main/).
 - A new options system unlocks future changes.
 
 ### Deprecations
 
 - **Python 2.7**: As announced in the v2.23.x release series, Pants v2.24 and later are not proactively tested in CI with Python 2.7 since [Python 2.7 is no longer supported by its maintainers as of 1 January 2020](https://www.python.org/doc/sunset-python-2/). While Pants may continue to work with Python 2.7 in the near term, Pants no longer officially supports use of Python 2.7, and, consequently, any remaining support for Python 2.7 may "bit rot" and diverge over time. Contributions to fix issues with Python 2.7 support will continue to be accepted, but will depend on any community contributions and will not constitute continued official support for Python 2.7.
 - **macOS versions**: Pants v2.24 is the last version that will support macOS 12, as macOS 12 is reaching the end of Apple's support. Future versions of Pants will require macOS 14 or newer (on arm64), and macOS 13 or newer (on x86-64). In addition, as announced in the v2.23.x release series, Pants v2.24 is built on macOS 12 and so may not work on versions of macOS 10.15 and 11 (which Apple no longer supports).
-
+- **Pants runner**: Pants v2.24 is the last version that supports versions of the `pants` launcher binary older than 0.12.2. Pants v2.25 and newer will require 0.12.2. [Follow the upgrade instructions](https://www.pantsbuild.org/2.24/docs/getting-started/installing-pants#upgrading-pants).
 
 ### General
 
@@ -58,19 +59,19 @@ Fix a bug where Pants sent a minor Go version instead of the major one to the Go
 
 Recognize `-fullpath` as a test binary flag.
 
-Add support for the `all:` prefix to patterns used with the `go:embed` directive. The `all:` prefix includes files which start with `_` or `.` which are ordinarilly excluded .
+Add support for the `all:` prefix to patterns used with the `go:embed` directive. The `all:` prefix includes files which start with `_` or `.` which are ordinarily excluded .
 
 #### JVM
 
 Compression of class files into a jar file is not reading or writing from the remote cache. The size of jar files, combined with the low computational cost of compression, can outweigh the advantages of using a remote cache.
 
-Fixed a coursier fetch wrapper script bug on nixos.
+Fixed a Coursier fetch wrapper script bug on NixOS.
 
-Fixed a jvm.sh script bug on nixos.
+Fixed a jvm.sh script bug on NixOS.
 
 #### Kotlin
 
-The kotlin linter, [ktlint](https://pinterest.github.io/ktlint/), has been updated to version 1.3.1.
+The Kotlin linter, [ktlint](https://pinterest.github.io/ktlint/), has been updated to version 1.3.1.
 
 #### Python
 
@@ -82,15 +83,15 @@ Pants' support for different Python versions has changed:
 
 - Pants v2.24 and later are not proactively tested in CI with Python 2.7 since [Python 2.7 is no longer supported by its maintainers as of 1 January 2020](https://www.python.org/doc/sunset-python-2/). While Pants may continue to work with Python 2.7 in the near term, Pants no longer officially supports use of Python 2.7, and, consequently, any remaining support for Python 2.7 may "bit rot" and diverge over time. Contributions to fix issues with Python 2.7 support will continue to be accepted, but will depend on any community contributions and will not constitute continued official support for Python 2.7.
 
-**Breaking change**: the `pants.backend.experimental.python.lint.ruff.check` and `pants.backend.experimental.python.lint.ruff.format` subsystems now execute Ruff as a downloaded binary, directly from the Ruff releases, rather than using PEX to execute the PyPI `ruff` package. This has less overhead, and makes customising the version simpler. However, **options like [`[ruff].install_from_resolve`](https://www.pantsbuild.org/2.24/reference/subsystems/ruff#install_from_resolve) are now ignored** and will be removed in future. If you have customised the version, use [`[ruff].version`](https://www.pantsbuild.org/2.24/reference/subsystems/ruff#version) and [`[ruff].known_versions`](https://www.pantsbuild.org/2.24/reference/subsystems/ruff#known_versions) instead. This version of Pants includes built-in known-versions for the latest patch release (at time of development) of each `0.x` minor series (0.7.2, 0.6.9, 0.5.7, 0.4.10, 0.3.7, 0.2.2, 0.1.15): if you install from a resolve with a particular pinned version of Ruff, you may find setting `[ruff].version` to the closest matching version in that list is acceptable (if not, you may need to add to `[ruff].known_versions`).
+**Breaking change**: the `pants.backend.experimental.python.lint.ruff.check` and `pants.backend.experimental.python.lint.ruff.format` subsystems now execute Ruff as a downloaded binary, directly from the Ruff releases, rather than using Pex to execute the PyPI `ruff` package. This has less overhead, and makes customizing the version simpler. However, **options like [`[ruff].install_from_resolve`](https://www.pantsbuild.org/2.24/reference/subsystems/ruff#install_from_resolve) are now ignored** and will be removed in future. If you have customized the version, use [`[ruff].version`](https://www.pantsbuild.org/2.24/reference/subsystems/ruff#version) and [`[ruff].known_versions`](https://www.pantsbuild.org/2.24/reference/subsystems/ruff#known_versions) instead. This version of Pants includes built-in known-versions for the latest patch release (at time of development) of each `0.x` minor series (0.7.2, 0.6.9, 0.5.7, 0.4.10, 0.3.7, 0.2.2, 0.1.15): if you install from a resolve with a particular pinned version of Ruff, you may find setting `[ruff].version` to the closest matching version in that list is acceptable (if not, you may need to add to `[ruff].known_versions`).
 
 Version Updates:
 
 - As a consequence of the lockfile generation, newer versions of many tools are now included in the default lockfiles.
 
-- The default version of pip is now [24.2](https://pip.pypa.io/en/stable/news/#v24-2) bringing performance improvements to dependency resolution and support or the upcoming Python 3.13 release.
+- The default version of pip is now [24.2](https://pip.pypa.io/en/stable/news/#v24-2) bringing performance improvements to dependency resolution and support for Python 3.13.
 
-- The versions of `setuptools` and `wheel` used in Pants have been upgraded to `74.1.2` and `0.44.0` to address a remote code execution vulnerability in versions before setuptools `70.0`. This forces the minimum Python version to 3.8, in line with the changes mentioned above.
+- The default versions of `setuptools` and `wheel` used in Pants have been upgraded to `74.1.2` and `0.44.0` to address a remote code execution vulnerability in versions before setuptools `70.0`. This forces the minimum Python version to 3.8, in line with the changes mentioned above.
 
 - The default version of the [Pex](https://docs.pex-tool.org/) tool has been updated from 2.16.2 to [2.20.3](https://github.com/pex-tool/pex/releases/tag/v2.20.3).
 
@@ -128,13 +129,13 @@ Added a new `cache_scope` field to [`adhoc_tool`](https://www.pantsbuild.org/2.2
 
 ### Plugin API changes
 
-The `path_metadata_request` intrinsic rule can now access metadata for paths in the local system outside of the build root. Use the new `namespace` field on `PathMetadataRequest` to request metdata on local system paths using namespace `PathNamespace.SYSTEM`.
+The `path_metadata_request` intrinsic rule can now access metadata for paths in the local system outside of the build root. Use the new `namespace` field on `PathMetadataRequest` to request metadata on local system paths using namespace `PathNamespace.SYSTEM`.
 
 PyO3, the interface crate between Rust and Python, has been upgraded to v0.22.x. A major change is that all Python values on the Rust side must be handled via either the `pyo3::Bound` or `pyo3::Py` smart pointers; direct references such as `&PyAny` are no longer supported.
 
 `GenerateToolLockfileSentinel` is removed. See the [porting guide details](https://www.pantsbuild.org/2.24/docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide#deprecated-generatetoollockfilesentinel) for instructions on migrating.
 
-Previously `SetupKwargs` took `_allow_banned_keys` which would allow one to pass in certain critical setuptools args (ex: `install_requires`) that Pants calculates for you.  If you Really Really know what you are doing you can know also use `_overwrite_banned_keys` to exclusively use your  own values and ignore the Pants calculated ones.
+Previously `SetupKwargs` took `_allow_banned_keys` which would allow one to pass in certain critical setuptools args (ex: `install_requires`) that Pants calculates for you.  If you Really Really know what you are doing you can know also use `_overwrite_banned_keys` to exclusively use your own values and ignore the Pants calculated ones.
 
 ## Full Changelog
 

docs/notes/2.25.x.md

@@ -2,10 +2,11 @@
 
 Pants is a fast, scalable, user-friendly build system for codebases of all sizes.
 
-Pants is an open-source project that is not owned or controlled by any one company or organization,
-and does incur some expenses. These expenses are managed by Pants Build, a non-profit that was
-established for this purpose. This non-profit's only source of revenue is
-[sponsorship](https://www.pantsbuild.org/sponsorship) by individuals and companies that use Pants.
+Pants is an open-source project that is not owned or controlled by any one company or organization, and does incur some expenses. These expenses are managed by Pants Build, a non-profit that was established for this purpose. This non-profit's only source of revenue is [sponsorship](https://www.pantsbuild.org/sponsorship) by individuals and companies that use Pants.
+
+We offer [formal sponsorship tiers for companies](https://www.pantsbuild.org/sponsorship), as well as individual sponsorships via [GitHub](https://github.com/sponsors/pantsbuild).
+
+Thank you to [Klayvio](https://www.klaviyo.com/) and [Normal Computing](https://normalcomputing.ai/) for their Platinum tier support through throughout this release.
 
 ## What's New
 
@@ -17,6 +18,7 @@ established for this purpose. This non-profit's only source of revenue is
 
 ### General
 
+- Fixed a longstanding bug in the processing of [synthetic targets](https://www.pantsbuild.org/2.24/docs/writing-plugins/the-target-api/concepts#synthetic-targets-api). This fix has the side-effect of requiring immutability and hashability of scalar values in BUILD files, which was always assumed but not enforced. This may cause BUILD file parsing errors, if you have custom field types involving custom mutable data structures. See ([#21725](https://github.com/pantsbuild/pants/pull/21725)) for more.
 - [Fixed](https://github.com/pantsbuild/pants/pull/21665) bug where `pants --export-resolve=<resolve> --export-py-generated-sources-in-resolve=<resolve>` fails (see [#21659](https://github.com/pantsbuild/pants/issues/21659) for more info).
 - [Fixed](https://github.com/pantsbuild/pants/pull/21694) bug where an `archive` target is unable to produce a ZIP file with no extension (see [#21693](https://github.com/pantsbuild/pants/issues/21693) for more info).
 
@@ -30,19 +32,14 @@ one in `remote_store_headers` or `remote_execution_headers`.
 
 The "legacy" options system is removed in this release. All options parsing is now handled by the new, native parser.
 
-
 ### Internal Python Upgrade
 
-The version of Python used by Pants itself has been updated to [3.11](https://docs.python.org/3/whatsnew/3.11.html). To support this the [Pants Launcher Binary](https://www.pantsbuild.org/blog/2023/02/23/the-pants-launcher-binary-a-much-simpler-way-to-install-and-run-pants) known as  [`scie-pants`](https://github.com/pantsbuild/scie-pants/) now has a minimum version of `0.12.0`.  To update to the latest launcher binary run:
-
-```
-SCIE_BOOT=update pants
-```
+The version of Python used by Pants itself has been updated to [3.11](https://docs.python.org/3/whatsnew/3.11.html). To support this the [Pants Launcher Binary](https://www.pantsbuild.org/blog/2023/02/23/the-pants-launcher-binary-a-much-simpler-way-to-install-and-run-pants) known as  [`scie-pants`](https://github.com/pantsbuild/scie-pants/) now has a minimum version of `0.12.2`.  To update to the latest launcher binary, either:
+- Use the package manager you used to install Pants. For example, with Homebrew: `brew update && brew upgrade pantsbuild/tap/pants`.
+- Use its built-in self-update functionality: `SCIE_BOOT=update pants`.
 
 That Pants itself happens to be partially writtin in Python has no bearing on the versions of Python that Pants can use to test and build your code.
 
-
-
 ### Goals
 
 ### Backends
@@ -57,6 +54,8 @@ Previously we did ad-hoc coercion of some field values, so that, e.g., you could
 Strict adherence to the [schema of Helm OCI registry configuration](https://www.pantsbuild.org/2.25/reference/subsystems/helm#registries) is now required.
 Previously we did ad-hoc coercion of some field values, so that, e.g., you could provide a "true"/"false" string as a boolean value. Now we require actual booleans.
 
+The `helm_infer.external_docker_images` glob syntax has been generalized.  In addition to `*`, you can now use Python [fnmatch](https://docs.python.org/3/library/fnmatch.html) to construct matterns like `quay.io/*`.
+
 #### Python
 
 The AWS Lambda backend now provides built-in complete platforms for the Python 3.13 runtime.

src/python/pants/backend/debian/target_types_test.py

@@ -28,17 +28,17 @@ def sources_rule_runner() -> RuleRunner:
 
 def test_sources_expected_num_files(sources_rule_runner: RuleRunner) -> None:
     sources_rule_runner.write_files(
-        {
-            f: ""
-            for f in [
+        dict.fromkeys(
+            [
                 "f1.txt",
                 "f2.txt",
                 "dirA/f3.txt",
                 "dirB/f4.txt",
                 "dirC/f5.txt",
                 "dirC/f6.txt",
-            ]
-        }
+            ],
+            "",
+        )
     )
 
     def hydrate(sources_cls: Type[DebianSources], sources: Iterable[str]) -> HydratedSources:

src/python/pants/backend/docker/goals/package_image.py

@@ -526,6 +526,8 @@ def parse_image_id_from_docker_build_output(docker: DockerBinary, *outputs: byte
                     r"(writing image (?P<digest>sha256:\S+))",
                     # BuildKit with containerd-snapshotter output.
                     r"(exporting manifest list (?P<manifest_list>sha256:\S+))",
+                    # BuildKit with containerd-snapshotter output and no attestation.
+                    r"(exporting manifest (?P<manifest>sha256:\S+))",
                     # Docker output.
                     r"(Successfully built (?P<short_id>\S+))",
                 ),
@@ -548,6 +550,7 @@ def parse_image_id_from_docker_build_output(docker: DockerBinary, *outputs: byte
                     image_id_match.group("digest")
                     or image_id_match.group("short_id")
                     or image_id_match.group("manifest_list")
+                    or image_id_match.group("manifest")
                 )
                 return image_id
 

src/python/pants/backend/docker/goals/package_image_test.py

@@ -1733,6 +1733,26 @@ def test_get_context_root(
             ),
             "",
         ),
+        # Buildkit with containerd-snapshotter 0.17.1 and disabled attestations
+        (
+            DockerBinary("/bin/docker", "1234", is_podman=False),
+            "sha256:6c3aff6414781126578b3e7b4a217682e89c616c0eac864d5b3ea7c87f1094d0",
+            dedent(
+                """\
+                    #24 exporting to image
+                    #24 exporting layers done
+                    #24 preparing layers for inline cache
+                    #24 preparing layers for inline cache 0.4s done
+                    #24 exporting manifest sha256:6c3aff6414781126578b3e7b4a217682e89c616c0eac864d5b3ea7c87f1094d0 0.0s done
+                    #24 exporting config sha256:af716170542d95134cb41b56e2dfea2c000b05b6fc4f440158ed9834ff96d1b4 0.0s done
+                    #24 naming to REDACTED:latest done
+                    #24 unpacking to REDACTED:latest 0.0s done
+                    #24 DONE 0.5s
+
+                    """
+            ),
+            "",
+        ),
         # Podman
         (
             DockerBinary("/bin/podman", "abcd", is_podman=True),

src/python/pants/backend/helm/dependency_inference/deployment.py

@@ -3,6 +3,7 @@
 
 from __future__ import annotations
 
+import fnmatch
 import logging
 import re
 from dataclasses import dataclass, field
@@ -256,17 +257,10 @@ def _image_ref_is_known_external(self, image_ref: str) -> bool:
         else:
             image_name = parsed.group("repository")
 
-        # Putting this wildcard check after parsing
-        # will mean that we don't approve things that don't look like docker images.
-        if "*" in self.helm_infer.external_docker_images:
-            return True
-        if (
-            image_name in self.helm_infer.external_docker_images
-            or image_ref in self.helm_infer.external_docker_images
-        ):
-            return True
-
-        return False
+        return any(
+            (fnmatch.fnmatch(image_name, pattern) or fnmatch.fnmatch(image_ref, pattern))
+            for pattern in self.helm_infer.external_docker_images
+        )
 
     def _handle_missing_docker_image(self, message):
         self.errors.append(message)