Test and release on newer macOS: 13 (x86-64) and 14 (arm64) #21655

huonw · 2024-11-17T23:03:31Z

This follows-up on the deprecation of support for older macOS versions that happened in 2.24.x and earlier, by switching main (2.25.x) to be building on macOS 13 for x86-64 (upgrading from 10.15) and macOS 14 for arm64 (upgrading from 11). These particular versions are dictated by Github's hosted runners: there's no arm64 macos-13 runners.

version	branch	supported (x86-64)	built-on (x86-64)	supported (arm64)	built-on (arm64)	refs
2.22.x	2.22.x	10.15	10.15 (self-hosted)	11	11 (self)	#21074
2.23.x	2.23.x	12	10.15 (self)	12	11 (self)	#21569 (also #21417 but reverted in #21623)
2.24.x	2.24.x	13	10.15 (self)	14	11 (self)	#21333, #21623
2.25.x	main	13	13 (Github hosted)	14	14 (GH)	#21333, #21413, this PR

This is preparation for scaling down our self-hosted macOS runners, but we cannot do that yet. They're still used by:

2.24.x branches and earlier (in this repo)
scie-pants' CI (CI uses pantsbuild's self-hosted macOS runners scie-pants#421)

Thus, we at least need to wait a few months for the 2.24.x milestone to be closed before we can reduce that spend.

(NB. despite the title, this doesn't start testing on arm64 macOS: see separate issue #20993.)

Fixes #21413
Fixes #21333

huonw · 2024-11-18T01:39:12Z

src/python/pants_release/generate_github_workflows.py

-GITHUB_HOSTED = {Platform.LINUX_X86_64, Platform.MACOS12_X86_64}
-SELF_HOSTED = {Platform.LINUX_ARM64, Platform.MACOS10_15_X86_64, Platform.MACOS11_ARM64}
-# We control these runners, so we preinstall and expose python on them.
-HAS_PYTHON = {Platform.LINUX_ARM64, Platform.MACOS10_15_X86_64, Platform.MACOS11_ARM64}


Moved to be implied by PYTHON_VERSIONS_PER_PLATFORM to manage the 3.7 & 3.8 availability.

huonw · 2024-11-18T01:39:25Z

src/python/pants_release/generate_github_workflows.py

-SELF_HOSTED = {Platform.LINUX_ARM64, Platform.MACOS10_15_X86_64, Platform.MACOS11_ARM64}
-# We control these runners, so we preinstall and expose python on them.
-HAS_PYTHON = {Platform.LINUX_ARM64, Platform.MACOS10_15_X86_64, Platform.MACOS11_ARM64}
-HAS_GO = {Platform.MACOS12_X86_64, Platform.MACOS10_15_X86_64, Platform.MACOS11_ARM64}


All platforms need Go explicitly installed now, it seems.

huonw · 2024-11-18T01:40:03Z

src/python/pants/base/exception_sink_test.py

@@ -55,6 +55,7 @@ def test_set_invalid_log_location():
        Platform.macos_x86_64: (
            "The provided log location path at '/' is not writable or could not be created: "
            "[Errno 21] Is a directory: '/'.",
+            "Error opening fatal error log streams for log location '/': [Errno 30] Read-only file system",


This matches Platform.macos_arm64 above: the test was failing on the macos-12 runner.

huonw · 2024-11-18T03:42:44Z

(I'm going to leave this one until after 2.25.0.dev0 is released, to reduce how many release-related things we're changing in one go / hopefully make it easier to bisect/debug if there's problems. #21528 and #21650 are already on main and waiting for the .dev0 release.)

cburroughs

I don't use macs, but all the yaml looks reasonable!

huonw · 2024-11-18T21:31:22Z

(I'm going to leave this one until after 2.25.0.dev0 is released, to reduce how many release-related things we're changing in one go / hopefully make it easier to bisect/debug if there's problems. #21528 and #21650 are already on main and waiting for the .dev0 release.)

Nevermind, this is blocking CI currently (including doing that release) due to brownouts: actions/runner-images#10721

So, I'll merge and we can just spend a bit more effort debugging in future, if we need to.

huonw · 2024-11-18T21:33:06Z

Ah, and I'll mark it for cherrypicking. We don't want to cherrypick the whole change, but at least the complete removal of the macOS-12 runners.

WorkerPants · 2024-11-18T21:34:21Z

I tried to automatically cherry-pick this change back to each relevant milestone, so that it is available in those older releases of Pants.

❌ 2.22.x

I was unable to cherry-pick this PR to 2.22.x, likely due to merge-conflicts.

Steps to Cherry-Pick locally

To resolve:

(Ensure your git working directory is clean)

Run the following script to reproduce the merge-conflicts:

git fetch https://github.com/pantsbuild/pants main \
  && git fetch https://github.com/pantsbuild/pants 2.22.x \
  && git checkout -b cherry-pick-21655-to-2.22.x FETCH_HEAD \
  && git cherry-pick 691d39619ac9e863d757b2cebcda71532ec6ae9c

Fix the merge conflicts and commit the changes
Run build-support/cherry_pick/make_pr.sh "21655" "2.22.x"

Please note that I cannot re-run CI if a job fails. Please work with your PR approver(s) to re-run CI if necessary.

❌ 2.23.x

I was unable to cherry-pick this PR to 2.23.x, likely due to merge-conflicts.

Steps to Cherry-Pick locally

To resolve:

(Ensure your git working directory is clean)

Run the following script to reproduce the merge-conflicts:

git fetch https://github.com/pantsbuild/pants main \
  && git fetch https://github.com/pantsbuild/pants 2.23.x \
  && git checkout -b cherry-pick-21655-to-2.23.x FETCH_HEAD \
  && git cherry-pick 691d39619ac9e863d757b2cebcda71532ec6ae9c

Fix the merge conflicts and commit the changes
Run build-support/cherry_pick/make_pr.sh "21655" "2.23.x"

Please note that I cannot re-run CI if a job fails. Please work with your PR approver(s) to re-run CI if necessary.

❌ 2.24.x

I was unable to cherry-pick this PR to 2.24.x, likely due to merge-conflicts.

Steps to Cherry-Pick locally

To resolve:

(Ensure your git working directory is clean)

Run the following script to reproduce the merge-conflicts:

git fetch https://github.com/pantsbuild/pants main \
  && git fetch https://github.com/pantsbuild/pants 2.24.x \
  && git checkout -b cherry-pick-21655-to-2.24.x FETCH_HEAD \
  && git cherry-pick 691d39619ac9e863d757b2cebcda71532ec6ae9c

Fix the merge conflicts and commit the changes
Run build-support/cherry_pick/make_pr.sh "21655" "2.24.x"

Please note that I cannot re-run CI if a job fails. Please work with your PR approver(s) to re-run CI if necessary.

When you're done manually cherry-picking, please remove the needs-cherrypick label on this PR.

Thanks again for your contributions!

🤖 Beep Boop here's my run link

#21655) (#21662) The macOS-12 runners are disappearing (see #21333), and thus our CI jobs that run on the macos-12 need to stop using it. This PR cherrypicks just that part of #21655: upgrading the test job, leaving the release jobs running on macOS 10.15 & 11.

#21655) (#21664) The macOS-12 runners are disappearing (see #21333), and thus our CI jobs that run on the macos-12 need to stop using it. This PR cherrypicks just that part of #21655: upgrading the test job, leaving the release jobs running on macOS 10.15 & 11.

#21655) (#21663) The macOS-12 runners are disappearing (see #21333), and thus our CI jobs that run on the macos-12 need to stop using it. This PR cherrypicks just that part of #21655: upgrading the test job, leaving the release jobs running on macOS 10.15 & 11.

Fixes #424 This makes a few adjustments to support running Pants 2.25.0.dev0 and newer, which run using Python 3.11 (pantsbuild/pants#21528), iterating on #351. Particularly: - expanding tools.pex's interpreter constraints to include 3.11 (and also 3.10, which seems fine) - add a test (NB. Pants 2.25 cannot run on macOS older than 13 / 14, depending on platform, so this test is conditional: pantsbuild/pants#21655) - hardcoding that 2.25.0.dev0 uses Python 3.11, to use the optimised "hit the exact URL first time" codepath, instead of having to try all Python versions: https://github.com/pantsbuild/scie-pants/blob/d50bd33fd67e944384c9548811a66c1cb03113a5/tools/src/scie_pants/pants_version.py#L239-L252 This thus preps scie-pants release 0.12.1 too, which I'll tag once merged.

huonw added 4 commits November 18, 2024 10:12

Test and release on macOS 13 (x86-64) and 14 (arm64)

6e47737

Delete old platforms

28cad91

Restrict Python versions on macOS arm64

e402b4c

release notes

640bc48

huonw force-pushed the huonw/macos-platform-support branch from a02995d to 640bc48 Compare November 17, 2024 23:33

huonw added 2 commits November 18, 2024 11:20

Install Go everywhere

a57ba0f

macOS12 tests

f04fa2f

huonw mentioned this pull request Nov 18, 2024

CI uses pantsbuild's self-hosted macOS runners pantsbuild/scie-pants#421

Open

huonw marked this pull request as ready for review November 18, 2024 01:37

huonw requested review from cburroughs, benjyw and tdyas November 18, 2024 01:37

huonw commented Nov 18, 2024

View reviewed changes

tdyas approved these changes Nov 18, 2024

View reviewed changes

huonw changed the title ~~Test and release on macOS 13 (x86-64) and 14 (arm64)~~ Test and release on newer macOS: 13 (x86-64) and 14 (arm64) Nov 18, 2024

cburroughs approved these changes Nov 18, 2024

View reviewed changes

huonw merged commit 691d396 into main Nov 18, 2024
24 checks passed

huonw deleted the huonw/macos-platform-support branch November 18, 2024 21:31

huonw added this to the 2.22.x milestone Nov 18, 2024

huonw added the needs-cherrypick label Nov 18, 2024

WorkerPants added the auto-cherry-picking-failed Auto Cherry-Picking Failed label Nov 18, 2024

huonw mentioned this pull request Nov 19, 2024

Test on newer macOS 13, not deprecated macOS 12 (partial cherry-pick of #21655) #21662

Merged

This was referenced Nov 19, 2024

Test on newer macOS 13, not deprecated macOS 12 (partial cherry-pick of #21655) #21663

Merged

Test on newer macOS 13, not deprecated macOS 12 (partial cherry-pick of #21655) #21664

Merged

huonw removed the needs-cherrypick label Nov 19, 2024

huonw mentioned this pull request Nov 25, 2024

Fix support for Pants 2.25.x / Python 3.11 pantsbuild/scie-pants#426

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test and release on newer macOS: 13 (x86-64) and 14 (arm64) #21655

Test and release on newer macOS: 13 (x86-64) and 14 (arm64) #21655

huonw commented Nov 17, 2024 •

edited

Loading

huonw Nov 18, 2024

huonw Nov 18, 2024

huonw Nov 18, 2024

huonw commented Nov 18, 2024

cburroughs left a comment

huonw commented Nov 18, 2024

huonw commented Nov 18, 2024

WorkerPants commented Nov 18, 2024

Test and release on newer macOS: 13 (x86-64) and 14 (arm64) #21655

Test and release on newer macOS: 13 (x86-64) and 14 (arm64) #21655

Conversation

huonw commented Nov 17, 2024 • edited Loading

huonw Nov 18, 2024

Choose a reason for hiding this comment

huonw Nov 18, 2024

Choose a reason for hiding this comment

huonw Nov 18, 2024

Choose a reason for hiding this comment

huonw commented Nov 18, 2024

cburroughs left a comment

Choose a reason for hiding this comment

huonw commented Nov 18, 2024

huonw commented Nov 18, 2024

WorkerPants commented Nov 18, 2024

❌ 2.22.x

❌ 2.23.x

❌ 2.24.x

huonw commented Nov 17, 2024 •

edited

Loading