filter known PBS release by the PBS-specific release tag #21710

tdyas · 2024-12-03T20:11:22Z

#21441 requests the ability to filter known versions of Python Build Standalone distribution by the PBS-specific release tag. This PR adds a new --python-build-standalone-release-constraints option to provide that filtering via a release constraint expression.

A constraint expression looks similar to Python build constraints; for example, >=20241201. Multiple constraints can be specified separated by commas; multiple constraints are AND'ed together.

The generator script for the known versions database has been updated to add a tag to each known version. The entire list as been regenerated.

Closes #21441.

tdyas

@cburroughs: Any thoughts on how user-supplied PBS versions should have the release tag specified?

tdyas · 2024-12-03T20:15:03Z

src/python/pants/backend/python/providers/python_build_standalone/constraints.py

@@ -0,0 +1,73 @@
+# Copyright 2024 Pants project contributors (see CONTRIBUTORS.md).


This file could probably be moved to more library-like location in the source tree. For now, I'll just leave it here since this is the only user.

tdyas · 2024-12-03T20:28:00Z

@cburroughs: Any thoughts on how user-supplied PBS versions should have the release tag specified?

We could probably parse it from the supplied URL. Of course, we could also probably parse out the Python version (which the user has to supply separately from the URL) as well at that point and not have the user specify it as they now must do.

cburroughs · 2024-12-04T15:22:14Z

@cburroughs: Any thoughts on how user-supplied PBS versions should have the release tag specified?

We could probably parse it from the supplied URL. Of course, we could also probably parse out the Python version (which the user has to supply separately from the URL) as well at that point and not have the user specify it as they now must do.

Hmm, like in the case of using the

3.1.2|macos_x86_64|6d0f18cd84b918c7b3edd0203e75569e0c7caecb1367bbbe409b44e28514f5be|42813|https://<URL>

syntax? I'm not sure. These could plausibly be file:// urls today or something else that doesn't have the same pattern at all? Maybe if you want to use the release tags you have to follow the same pattern as upstream and parse it out of the url.

Some regex prior art: astral-sh/python-build-standalone#127 (comment)

tdyas · 2024-12-04T15:28:32Z

syntax? I'm not sure. These could plausibly be file:// urls today or something else that doesn't have the same pattern at all? Maybe if you want to use the release tags you have to follow the same pattern as upstream and parse it out of the url.

The parsing could be best effort. If we can extract a release tag from the URL then we use it and match against it. If not, then we just do not apply the release tag filter to that PBS release (i.e., the filter always includes the release).

cburroughs

I played around with this in example-python and verified I got different openssl versions (aka different PBS releases) so the basic structure looks good. I'm not sure how best to handle "unsatisfied" constraints. Like if one asks for something that does not exist like:

[python-build-standalone-python-provider]
release_constraints="==20240000"

It isn't an error, which may be surprising.

src/python/pants/backend/python/providers/python_build_standalone/rules.py

tdyas · 2024-12-04T16:17:15Z

I played around with this in example-python and verified I got different openssl versions (aka different PBS releases) so the basic structure looks good. I'm not sure how best to handle "unsatisfied" constraints. Like if one asks for something that does not exist like:
[python-build-standalone-python-provider]
release_constraints="==20240000"
It isn't an error, which may be surprising.

If there are no user-supplied versions, then this should error since it should find no matching PBS releases (but will add a test for this behavior). But yeah, if we "accept" any user-supplied version without a parseable release tag, then non-conforming versions would be accepted.

Maybe Pants should warn or error (configurable?) if it cannot parse the release tag out of a PBS release URL?

cburroughs · 2024-12-05T01:26:22Z

Maybe Pants should warn or error (configurable?) if it cannot parse the release tag out of a PBS release URL?

I think that would be consistent with most of the other Pants warn/error configuration knobs.

Although FWIW in this case I was messing around with the bundled json encoded versions, so I don't think that would be the cause of the ==does-not-exist issue.

tdyas · 2024-12-09T19:17:45Z

f there are no user-supplied versions, then this should error since it should find no matching PBS releases (but will add a test for this behavior). But yeah, if we "accept" any user-supplied version without a parseable release tag, then non-conforming versions would be accepted.

Some of the PBS versions in versions_info.json did not have tag data added by the script so they were "accepted" as conforming. I will filter them out for now.

tdyas · 2024-12-09T19:32:15Z

Latest commits makes sure all known versions have a tag to match against.

tdyas · 2024-12-09T20:52:55Z

Inference of the release tag is implemented in #21739 (which stacks on top of this PR).

huonw

Thanks for improving the PBS provider!

docs/notes/2.25.x.md

src/python/pants/backend/python/providers/python_build_standalone/constraints.py

src/python/pants/backend/python/providers/python_build_standalone/rules.py

src/python/pants/backend/python/providers/python_build_standalone/constraints.py

src/python/pants/backend/python/providers/python_build_standalone/rules.py

Co-authored-by: Huon Wilson <[email protected]>

…one/constraints.py Co-authored-by: Huon Wilson <[email protected]>

Co-authored-by: Huon Wilson <[email protected]>

tdyas · 2024-12-10T17:22:20Z

Thanks for improving the PBS provider!

@huonw: I've updated the PR based on your comments. Please let me know if there is anything else!

huonw

The code itself looks great, but I have two potential bigger-picture things:

The PBS versions Pants has scraped are now explicitly part of the 'public API'.
We currently only record one release tag for each python version, so release_constraints are somewhat redundant with interpreter_constraints.

I think the current generate_urls.py script isn't super-careful about how it records assert versions: it looks like it seems to only include the most recent release for each version(?).

This may lead to Pants scraping new versions being a breaking change for some users.

For instance, it seems all of these releases include 3.12.7:

If we had scraped on 2024-10-10, we would record the 20241008 assets only (e.g. asset URLs like https://github.com/indygreg/python-build-standalone/releases/download/20241008/cpython-3.12.7%2B20241008-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz).

Thus, someone using Pants 2.24 might write interpreter_constraints == ["==3.12.7"], release_constraints = "==20241008" (e.g. they identified that 20241002 and 20241016 had bugs).

Then, a while later, we re-run the scraping, which picks up 20241016 and thus Pants 2.25 only has those assets (e.g. https://github.com/indygreg/python-build-standalone/releases/download/20241016/cpython-3.12.7%2B20241016-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz).

The user upgrading from Pants 2.24 to Pants 2.25 will experience breakage because release_constraints = "==20241008" will no longer be satisfied: there's no longer 3.12.7 assets with that tag.

To solve this, I think we can record more assets into versions_info.json: every single release for every single version.

However, I think this can and should be a follow-up issue and follow-up work building on the groundwork done in this PR: this provider is experimental for a reason! We don't need to get every change perfect before landing.

tdyas · 2024-12-11T15:36:34Z

To solve this, I think we can record more assets into versions_info.json: every single release for every single version.

I agree. Good points about the problems which ensue if all of the release data is not recorded.

huonw · 2024-12-11T20:54:02Z

I filed #21748

@huonw

…21776) As first mentioned by @huonw in #21710 (review) and more fully discussed in #21748, the addition of filtering PBS releases by the PBS release tag means that users may now see a breaking change if Pants scrapes newer versions of PBS releases, replaces some of the existing "known versions" data with the newly-scraped release metadata, and the user selects an older release via a PBS release constraint. In that case, the user would see an error because Pants would be unable to select that specific PBS version anymore due to no longer having any metadata regarding that PBS release tag. The solution is for Pants to store metadata for _all_ PBS release tags. Newer versions will only add to the PBS "known versions" data and not replace nor delete any metadata. With all metadata available, a user who selects a particular PBS release via the release constraints will still continue to see that release be selected. And users who want the latest matching PBS release will continue to get the latest PBS release (known to Pants). Closes #21748.

tdyas and others added 6 commits November 21, 2024 09:59

checkpoint

21513e8

Merge branch 'main' into pbs/release-constraint

f82450c

checkpoint

ae1de50

bug fixs and tests

cf94701

do not assume tag is set

b04b1a0

release notes

205ba3e

tdyas added the category:new feature label Dec 3, 2024

tdyas requested a review from cburroughs December 3, 2024 20:14

tdyas commented Dec 3, 2024

View reviewed changes

cburroughs reviewed Dec 4, 2024

View reviewed changes

src/python/pants/backend/python/providers/python_build_standalone/rules.py Outdated Show resolved Hide resolved

Merge branch 'main' into pbs/release-constraint

a995605

Tom Dyas added 2 commits December 9, 2024 14:27

filter out PBS releases with no tag metadata

4a7d348

improve help text for release constraints option

4a3e038

tdyas marked this pull request as ready for review December 9, 2024 19:32

tdyas requested review from benjyw and huonw December 9, 2024 19:32

do not set tag for user-supplied releases

79b7f73

tdyas mentioned this pull request Dec 9, 2024

infer the PBS release tag from the user-supplied URLs #21739

Merged

empty string - mypy

345eacd

huonw reviewed Dec 9, 2024

View reviewed changes

tdyas and others added 2 commits December 9, 2024 18:16

Update docs/notes/2.25.x.md

9c5c8db

Co-authored-by: Huon Wilson <[email protected]>

Update src/python/pants/backend/python/providers/python_build_standal…

5474419

…one/constraints.py Co-authored-by: Huon Wilson <[email protected]>

identify option causing error

69e2aee

Co-authored-by: Huon Wilson <[email protected]>

tdyas force-pushed the pbs/release-constraint branch from c65b0d7 to 69e2aee Compare December 10, 2024 16:36

Tom Dyas added 3 commits December 10, 2024 11:46

rename evaluate to is_satisfied and add protocol

d156f07

use str.removeprefix in parsing constraints to simplify

71f8c32

more idiomatic code

9996439

huonw approved these changes Dec 10, 2024

View reviewed changes

tdyas merged commit 2ad1796 into pantsbuild:main Dec 11, 2024
24 checks passed

tdyas deleted the pbs/release-constraint branch December 11, 2024 15:36

huonw mentioned this pull request Dec 11, 2024

PBS python provider release_constraints needs fine-tuning to avoid back-compat risk #21748

Closed

tdyas mentioned this pull request Dec 18, 2024

store all known releases for Python Build Standalone known versions #21776

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filter known PBS release by the PBS-specific release tag #21710

filter known PBS release by the PBS-specific release tag #21710

tdyas commented Dec 3, 2024 •

edited

Loading

tdyas left a comment

tdyas Dec 3, 2024

tdyas commented Dec 3, 2024 •

edited

Loading

cburroughs commented Dec 4, 2024

tdyas commented Dec 4, 2024

cburroughs left a comment

tdyas commented Dec 4, 2024 •

edited

Loading

cburroughs commented Dec 5, 2024

tdyas commented Dec 9, 2024

tdyas commented Dec 9, 2024

tdyas commented Dec 9, 2024

huonw left a comment

tdyas commented Dec 10, 2024 •

edited

Loading

huonw left a comment

tdyas commented Dec 11, 2024

huonw commented Dec 11, 2024

		@@ -0,0 +1,73 @@
		# Copyright 2024 Pants project contributors (see CONTRIBUTORS.md).

filter known PBS release by the PBS-specific release tag #21710

filter known PBS release by the PBS-specific release tag #21710

Conversation

tdyas commented Dec 3, 2024 • edited Loading

tdyas left a comment

Choose a reason for hiding this comment

tdyas Dec 3, 2024

Choose a reason for hiding this comment

tdyas commented Dec 3, 2024 • edited Loading

cburroughs commented Dec 4, 2024

tdyas commented Dec 4, 2024

cburroughs left a comment

Choose a reason for hiding this comment

tdyas commented Dec 4, 2024 • edited Loading

cburroughs commented Dec 5, 2024

tdyas commented Dec 9, 2024

tdyas commented Dec 9, 2024

tdyas commented Dec 9, 2024

huonw left a comment

Choose a reason for hiding this comment

tdyas commented Dec 10, 2024 • edited Loading

huonw left a comment

Choose a reason for hiding this comment

tdyas commented Dec 11, 2024

huonw commented Dec 11, 2024

tdyas commented Dec 3, 2024 •

edited

Loading

tdyas commented Dec 3, 2024 •

edited

Loading

tdyas commented Dec 4, 2024 •

edited

Loading

tdyas commented Dec 10, 2024 •

edited

Loading