Support uploading windows and linux aarch64 wheels

[mroeschke]

Over in pandas we're trying to use this action to upload to https://anaconda.org/channels/scientific-python-nightly-wheels/packages/pandas/overview and the following platforms are failing (example failing job: https://github.com/pandas-dev/pandas/actions/runs/20086600506)

• win-64
• win-arm64
• manylinux-aarch64
• musllinux-aarch64

I think these are largely failing because these platforms are not specified in the platforms in the pixi.toml

upload-nightly-action/pixi.toml

Line 6 in 64b7bfd

platforms = ["linux-64", "osx-64", "osx-arm64"]

[matthewfeickert]

You're correct, though the assumption is that most people are collating builds and uploading them in a single job (e.g. Matplotlib's workflow).

Your PR is probably reasonable to support people however they would like, but is there a reason you're uploading individually, instead of uploading only on verified success of all builds?

[mroeschke]

Our wheel GHA workflow is to create a matrix over each platform to build then test then upload if tests pass since the wheel is already on that host https://github.com/pandas-dev/pandas/blob/main/.github/workflows/wheels.yml#L81. We could do that upload/download dance that Matplotlib does I suppose.

is there a reason you're uploading individually, instead of uploading only on verified success of all builds?

Mostly for the convenience for the wheel being available on the host after tests pass already. Idk if there is "nightly wheel etiquette" of all platforms having tests passing before uploading all their wheels, but since they are development versions IMO it's OK for 2 nightlies to possibly be slightly different

[matthewfeickert]

but since they are development versions IMO it's OK for 2 nightlies to possibly be slightly different

This seems not ideal to me. The point of the nightly wheels is to provide them for community testing use. Having, for example, Linux tests pass and Windows tests fail because you're getting different builds seems anti-helpful.

[matthewfeickert]

... and the following platforms are failing (example failing job: https://github.com/pandas-dev/pandas/actions/runs/20086600506)
...

• win-arm64

Given the lack of win-arm64 supported feedstocks on conda-forge (c.f. #151 (review)) you will need to do some consolidation of your upload process to get the win-arm64 wheels uploaded.

[betatim]

I came here to learn why adding more platforms to the pixi config was the answer to supporting uploading wheels for those platforms. Now that I've got my answer I feel sucked into the discussion about requiring tests to pass or not :-/

I'm not sure that (the full suite of) tests need to pass before uploading a wheel. There are many reasons tests fail that has nothing to do with the wheel being broken (👋 flaky tests). So I am more and more convinced that you should upload things no matter what the tests say, assuming some very basic import my_library and my_library.do_something_very_basic_that_shows_the_build_worked_and_didnt_crash() style tests pass.

One thing that could be confusing is if there are nightlies with different versions for different platforms. At least I am pretty confident it would take me a long time to think to check for this when trying to understand why some of my project's nightly tests fail and others didn't. Maybe this is a small reason to have a "all or nothing" upload style (because we don't immediately delete the previous nightly.)

[matthewfeickert]

One thing that could be confusing is if there are nightlies with different versions for different platforms. At least I am pretty confident it would take me a long time to think to check for this when trying to understand why some of my project's nightly tests fail and others didn't.

This is my main concern (sorry, I guess I didn't do a good job of making this clear).

[Carreau]

I think if we have nightly for end users, it's fine to have different upload time/slightly different versions; though one of the goal of this channel is nightly for pre-release CI for downstream consumers, and in that case, I think we aught to have passing and identical wheels so that if a job fail downstream maintainers are not going to scratch their head too much.

Though this is not a novel problem, there is a proposal for two state release on PyPI to make sure a new release is atomic:

pypi/warehouse#726

Because if all the wheels are on the same host and something goes down during the wheel upload, you can still get an inconsistent state.

[mroeschke]

Happy to follow any conventions about the state of the nightly wheel (e.g. able to install, able to pass tests, etc) that other projects use. The arguments about ensuring all wheels are in sync relative to the latest git commit makes sense