Upgrade the default version of black from 23.12 to 24.8 #21056

huonw · 2024-06-13T00:33:23Z

This upgrades the default black version from 23.3.0 to 24.8.0.

This results in "user-visible" formatting changes: we're ticking over into a new year, and thus pick up the "2024" stable style in 24.1.0: https://black.readthedocs.io/en/stable/change_log.html#id18 . Depending on the codebase, this potentially results in significant reformatting (and does so for Pants).

To make it easier for people to upgrade, this retains a copy of the old lockfile, that can be referenced via a resources: URI.

There's various other changes in the release notes, like support for new syntax, bug fixes and performance improvements.

Because Pants uses its own black backend this PR also reformats the world, according to the new style.

The commits are independently sensible:

Upgrade black itself
Adjust the flake8 rules due to a new conflict (https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html#e701-e704)
Reformat the whole world
Various fixes after review

huonw · 2024-06-13T00:35:49Z

docs/notes/2.23.x.md

@@ -60,6 +60,8 @@ The backend linter will also load a Trufflehog [configuration file](https://gith

 #### Python

+[The `pants.backend.python.lint.black` backend](https://www.pantsbuild.org/2.23/reference/subsystems/black) now uses version 24.4.2 by default, upgrading from 23.3.0. This comes with two notable visible changes: there's a new stable style (see [release notes for 24.1.0](https://black.readthedocs.io/en/stable/change_log.html) for details) which may result in extensive reformatting, and black now requires Python 3.8 or newer to _run_ (but can still format code from older versions). To override Pants' default version, use [the `install_from_resolve` option](https://www.pantsbuild.org/2.23/reference/subsystems/black#install_from_resolve) and/or [the `interpreter_constraints` option](https://www.pantsbuild.org/2.23/reference/subsystems/black#interpreter_constraints).


(see [release notes for 24.1.0](https://black.readthedocs.io/en/stable/change_log.html) for details)

NB. this doesn't link to the release notes heading directly because they're just numbered links from the top, not permalinks, e.g. https://black.readthedocs.io/en/stable/change_log.html#id18

This means a direct link will be wrong/misleading by the time we release Pants 2.23, if black does any releases in that time, since the 18th heading won't be 24.1.0 any more.

psf/black#4344

benjyw · 2024-06-13T03:39:56Z

This is very disruptive to users - they upgrade Pants and suddenly all their code reformats...

I guess that is just how Black is, and we should go along with it?

huonw · 2024-06-13T04:13:38Z

Yes, agreed. I was trying to alleviate some of the issue with the release notes call out, but we might want to do more leg work.

As you say, this is how black is, and I don't think pants should be imposing its own ideas on tools, thus getting stuck on really old versions...

Just brainstorming some ideas:

build infrastructure for Python tools similar to "known versions" for external tools, so Pants could have several built-in lockfiles that can be chosen between more simply than creating a whole custom resolve
turn Black into an external tool somehow, e.g. have a parallel repo that publishes pexes for it
have separate backends for each version of black, e.g. a pants.backend.python.lint.black23, pants.backend.python.lint.black24 then black25, with the current pants.backend.python.lint.black => pants.backend.python.lint.black23
more visible warnings/callouts about the change than just release notes
maybe you can think of other things?

I'm inclined to either 1 or 4, because as you say, that's just how black is... Thoughts?

lilatomic · 2024-06-13T04:38:25Z

I think that if we shipped both lockfiles, we could make rolling back be as simple as creating a resolve that points to the lockfile using the internal syntax (resource://), something like

[python.resolve]
black = "resource://pants/backend/python/lint/black/black23.lock"
[black]
install_from_resolve = "black"

kaos · 2024-06-13T11:53:56Z

It would be nice if it was less involved to override which version of black to use. i.e. provide the version and pants takes care of the locking and what-not behind the scenes. But I think that is for another issue, and here, the disruption comes from black, and I'd say we don't want to get stuck on an increasingly old version for the sake of shielding users from that. Also, there is a way out of it--so we're not imposing anything impossible.

I like @lilatomic's idea as middle ground to offer an alternative built-in lockfile. But I don't know how well that plays with the change in ICs.. or if it's enough to set those along with the lockfile in the config?

benjyw

Yeah, agreed. Live by the tool, die by the tool...

huonw · 2024-06-13T23:52:31Z

I think that if we shipped both lockfiles, we could make rolling back be as simple as creating a resolve that points to the lockfile using the internal syntax (resource://), something like

Nice idea, I'll have a play with that and see if it seems like a viable "ease the transition" path.

huonw · 2024-06-14T08:51:59Z

Nice, that seems to work well, running the following config (with PANTS_SOURCE=... on this branch) uses the old 23.3 version and the thus the old formatting style.

[GLOBAL]
pants_version = "2.21.0"
backend_packages = ["pants.backend.python", "pants.backend.python.lint.black"]

[python]
interpreter_constraints = ["==3.10.*"]
enable_resolves = true

[python.resolves]
black = "resource://pants.backend.python.lint.black/black-23.3.lock"

[black]
install_from_resolve = "black"

I've pushed this and rewritten the release notes to explain the upgrade path.

I'm thinking we could then deprecate this lockfile in Pants 2.24, and remove in Pants 2.26 (or something like that).

What do you think? Is it worth it?

benjyw · 2024-06-15T03:33:10Z

I think we can even deprecate in 2.23, and remove in 2.24. That would be more in the spirit of Black's apparent upgrade strategy of "just disrupt the world". But we don't need to deprecate the lockfile's existence, just the relying on it by default?

huonw · 2024-06-16T23:04:42Z

Hm, I'm not 100% sure I understand your idea. Are you suggesting this?

In pants 2.23:
- still use black 23 by default, but emits deprecation warnings
- we encourage users to (explicitly) switch to black 24 somehow, potentially with a resources://.../black24.lock lockfile packaged
In pants 2.24:
- switch to black 24 by default
- retain the black 23 lockfile for a user to use explicitly, if required

Or maybe something else?

benjyw · 2024-06-17T15:25:18Z

Hm, I'm not 100% sure I understand your idea. Are you suggesting this?

In pants 2.23:

still use black 23 by default, but emits deprecation warnings

we encourage users to (explicitly) switch to black 24 somehow, potentially with a resources://.../black24.lock lockfile packaged

In pants 2.24:

switch to black 24 by default

retain the black 23 lockfile for a user to use explicitly, if required

Or maybe something else?

Yes, this. Or we can get rid of the black 23 lockfile if we prefer. I'm just saying that we don't have to.

huonw · 2024-09-11T21:29:00Z

We've just branched for 2.23, so merging this pull request now will come out in 2.24, please move the release notes updates to docs/notes/2.24.x.md. Thank you!

cburroughs · 2024-12-17T18:39:54Z

@huonw At this point, are you leaning towards trying to land this in 2.25?

huonw · 2024-12-18T00:14:14Z

Ah, hm, I lost track of this one. I'll get it refreshed in the next few days.

huonw · 2024-12-19T08:30:39Z

I've updated this PR and written a (probably too large) upgrade guide in the release notes. Suggestions for how to cut it down most welcome!

Thanks for the prompt

cburroughs

Thanks! Left some minor optional comments.

cburroughs · 2024-12-20T19:06:52Z

build-support/bin/generate_builtin_lockfiles.py

@@ -89,13 +89,11 @@ def lockfile_name(self) -> str:


 @dataclass
-class PythonTool(Tool[PythonToolRequirementsBase]):
-    ...
+class PythonTool(Tool[PythonToolRequirementsBase]): ...


Tangential rant: Oh that's rather ugly, but I guess that's on brand for black.

cburroughs · 2024-12-20T19:08:29Z

docs/notes/2.25.x.md

+
+[The `pants.backend.python.lint.black` backend](https://www.pantsbuild.org/2.25/reference/subsystems/black) now uses version 24.8.0 by default, upgrading from 23.12.1. This comes with a new stable style (see [release notes for 24.1.0](https://black.readthedocs.io/en/stable/change_log.html) for details) which may result in extensive reformatting.
+
+To override Pants' default version, use [the `install_from_resolve` option](https://www.pantsbuild.org/2.25/reference/subsystems/black#install_from_resolve) and/or [the `interpreter_constraints` option](https://www.pantsbuild.org/2.25/reference/subsystems/black#interpreter_constraints). The style changes may be extensive, so, to make upgrading to Pants 2.25 easier, we provide the old lockfile as built-in, for now (Pants may remove this in future, so you should either switch to your own lockfile, or upgrade to the default Black 24). To use this lockfile, and remain on Black 23.3, configure a resolve as follows:


Suggested change

To override Pants' default version, use [the `install_from_resolve` option](https://www.pantsbuild.org/2.25/reference/subsystems/black#install_from_resolve) and/or [the `interpreter_constraints` option](https://www.pantsbuild.org/2.25/reference/subsystems/black#interpreter_constraints). The style changes may be extensive, so, to make upgrading to Pants 2.25 easier, we provide the old lockfile as built-in, for now (Pants may remove this in future, so you should either switch to your own lockfile, or upgrade to the default Black 24). To use this lockfile, and remain on Black 23.3, configure a resolve as follows:

To override Pants' default version, use [the `install_from_resolve` option](https://www.pantsbuild.org/2.25/reference/subsystems/black#install_from_resolve) and/or [the `interpreter_constraints` option](https://www.pantsbuild.org/2.25/reference/subsystems/black#interpreter_constraints). The style changes may be extensive, so, to make upgrading to Pants 2.25 easier, we provide the old lockfile as built-in, for now (Pants remove this in a future release, so you should either switch to your own lockfile, or upgrade to the default Black 24). To use this lockfile, and remain on Black 23.3, configure a resolve as follows:

At the limit eventually the CPython version compatible with the old black version will be EoL. We can't support it forewever.

cburroughs · 2024-12-20T19:12:21Z

docs/notes/2.25.x.md

+install_from_resolve = "black"
+```
+
+To take control of your Black version independent of Pants' default, configure a resolve similar to the following, and generate the lockfile with `pants generate-lockfiles --resolve=black`:


Suggested change

To take control of your Black version independent of Pants' default, configure a resolve similar to the following, and generate the lockfile with `pants generate-lockfiles --resolve=black`:

To take control of your Black version independent of Pants' default, configure a resolve similar to the following, and generate the lockfile with `pants generate-lockfiles --resolve=your-resolve-name`:

I waffle on this, not sure what makes for the better example naming.

huonw added the category:user api change label Jun 13, 2024

huonw commented Jun 13, 2024

View reviewed changes

huonw marked this pull request as draft June 13, 2024 00:36

huonw added 3 commits June 13, 2024 10:48

Upgrade black to 24.4.2

44b019e

Ignore E701 in flake8

03acbbb

Reformat the world

a38eaaf

huonw force-pushed the huonw/upgrade-black branch from cdf52c6 to a38eaaf Compare June 13, 2024 00:48

huonw marked this pull request as ready for review June 13, 2024 01:10

huonw requested review from kaos, benjyw and thejcannon June 13, 2024 01:13

kaos approved these changes Jun 13, 2024

View reviewed changes

benjyw approved these changes Jun 13, 2024

View reviewed changes

huonw mentioned this pull request Jun 14, 2024

Upgrade IPython 7.34.0 -> 8.12.3, dropping py3.7 support #21059

Closed

huonw added 2 commits June 14, 2024 18:41

Add old lockfile as a "backup" resource

cc1a38e

Move notes

4d7cbee

kaos approved these changes Jun 14, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/main' into huonw/upgrade-black

f590313

huonw added 5 commits December 19, 2024 12:15

Switch old lockfile to 23.12

3e65070

Update built-in lockfile to 24.8.0

0a25246

2.25 release notes

707245f

BUILD file dep

c6aac0e

reformat new code

75b884e

huonw changed the title ~~Upgrade the default version of black from 23.3.0 to 24.4.2~~ Upgrade the default version of black from 23.12 to 24.8 Dec 19, 2024

huonw requested a review from cburroughs December 19, 2024 08:30

cburroughs approved these changes Dec 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade the default version of black from 23.12 to 24.8 #21056

Upgrade the default version of black from 23.12 to 24.8 #21056

huonw commented Jun 13, 2024 •

edited

Loading

huonw Jun 13, 2024

benjyw commented Jun 13, 2024

huonw commented Jun 13, 2024 •

edited

Loading

lilatomic commented Jun 13, 2024

kaos commented Jun 13, 2024

benjyw left a comment

huonw commented Jun 13, 2024

huonw commented Jun 14, 2024 •

edited

Loading

benjyw commented Jun 15, 2024

huonw commented Jun 16, 2024

benjyw commented Jun 17, 2024

huonw commented Sep 11, 2024

cburroughs commented Dec 17, 2024

huonw commented Dec 18, 2024

huonw commented Dec 19, 2024

cburroughs left a comment

cburroughs Dec 20, 2024

cburroughs Dec 20, 2024

cburroughs Dec 20, 2024

cburroughs Dec 20, 2024

cburroughs Dec 20, 2024

		@@ -60,6 +60,8 @@ The backend linter will also load a Trufflehog [configuration file](https://gith

		#### Python

		[The `pants.backend.python.lint.black` backend](https://www.pantsbuild.org/2.23/reference/subsystems/black) now uses version 24.4.2 by default, upgrading from 23.3.0. This comes with two notable visible changes: there's a new stable style (see [release notes for 24.1.0](https://black.readthedocs.io/en/stable/change_log.html) for details) which may result in extensive reformatting, and black now requires Python 3.8 or newer to _run_ (but can still format code from older versions). To override Pants' default version, use [the `install_from_resolve` option](https://www.pantsbuild.org/2.23/reference/subsystems/black#install_from_resolve) and/or [the `interpreter_constraints` option](https://www.pantsbuild.org/2.23/reference/subsystems/black#interpreter_constraints).


		[The `pants.backend.python.lint.black` backend](https://www.pantsbuild.org/2.25/reference/subsystems/black) now uses version 24.8.0 by default, upgrading from 23.12.1. This comes with a new stable style (see [release notes for 24.1.0](https://black.readthedocs.io/en/stable/change_log.html) for details) which may result in extensive reformatting.

		To override Pants' default version, use [the `install_from_resolve` option](https://www.pantsbuild.org/2.25/reference/subsystems/black#install_from_resolve) and/or [the `interpreter_constraints` option](https://www.pantsbuild.org/2.25/reference/subsystems/black#interpreter_constraints). The style changes may be extensive, so, to make upgrading to Pants 2.25 easier, we provide the old lockfile as built-in, for now (Pants may remove this in future, so you should either switch to your own lockfile, or upgrade to the default Black 24). To use this lockfile, and remain on Black 23.3, configure a resolve as follows:

	To take control of your Black version independent of Pants' default, configure a resolve similar to the following, and generate the lockfile with `pants generate-lockfiles --resolve=black`:
	To take control of your Black version independent of Pants' default, configure a resolve similar to the following, and generate the lockfile with `pants generate-lockfiles --resolve=your-resolve-name`:

Upgrade the default version of black from 23.12 to 24.8 #21056

Are you sure you want to change the base?

Upgrade the default version of black from 23.12 to 24.8 #21056

Conversation

huonw commented Jun 13, 2024 • edited Loading

huonw Jun 13, 2024

Choose a reason for hiding this comment

benjyw commented Jun 13, 2024

huonw commented Jun 13, 2024 • edited Loading

lilatomic commented Jun 13, 2024

kaos commented Jun 13, 2024

benjyw left a comment

Choose a reason for hiding this comment

huonw commented Jun 13, 2024

huonw commented Jun 14, 2024 • edited Loading

benjyw commented Jun 15, 2024

huonw commented Jun 16, 2024

benjyw commented Jun 17, 2024

huonw commented Sep 11, 2024

cburroughs commented Dec 17, 2024

huonw commented Dec 18, 2024

huonw commented Dec 19, 2024

cburroughs left a comment

Choose a reason for hiding this comment

cburroughs Dec 20, 2024

Choose a reason for hiding this comment

cburroughs Dec 20, 2024

Choose a reason for hiding this comment

cburroughs Dec 20, 2024

Choose a reason for hiding this comment

cburroughs Dec 20, 2024

Choose a reason for hiding this comment

cburroughs Dec 20, 2024

Choose a reason for hiding this comment

huonw commented Jun 13, 2024 •

edited

Loading

huonw commented Jun 13, 2024 •

edited

Loading

huonw commented Jun 14, 2024 •

edited

Loading