[Python][Packaging] Update python_wheel_windows_build.bat to use released delvewheel

[raulcd]

Describe the enhancement requested

Delvewheel 1.10 has been released with the fix to --with-mangle we required here:

• GH-44855: [Python][Packaging] Use delvewheel to repair Windows wheels #35323

We should be able to remove installing delvewheel from a github branch and just use the new version.

arrow/ci/scripts/python_wheel_windows_build.bat

Lines 142 to 150 in 4ede48c

	@REM Repair the wheel with delvewheel
	@REM
	@REM Since we bundled the Arrow C++ libraries ourselves, we only need to
	@REM mangle msvcp140.dll so as to avoid ABI issues when msvcp140.dll is
	@REM required by multiple Python libraries in the same process.
	@REM
	@REM For now this requires a custom version of delvewheel:
	@REM https://github.com/adang1345/delvewheel/pull/59
	%PYTHON_CMD% -m pip install https://github.com/pitrou/delvewheel/archive/refs/heads/fixes-for-arrow.zip || exit /B 1

Component(s)

Python, Packaging

[raulcd]

To have in mind that the original PR wasn't merged:
https://github.com/adang1345/delvewheel/pull/59/files
and the author created a slightly different API:
adang1345/delvewheel@0bc542d

which I haven't investigated if it does exactly the same but the original issue was closed:

• No mangling occurs adang1345/delvewheel#58

[protokoul]

Hello. I would like to contribute here. Could you please help me to understand how can I test my changes in .bat file?

[raulcd]

Hi @protokoul ,

Sure, thanks for taking a look. I added the good-first-issue label before realizing the change on delvewheel wasn't really the PR that was provided but a different one with a slightly different API so it's slightly more complex than I initially anticipated as you'll have to also update the delvewheel command used.

If you have windows and docker you can try using the docker compose image: python-wheel-windows-vs2019 here:

arrow/docker-compose.yml

Lines 1286 to 1300 in 4ede48c

	python-wheel-windows-vs2019:
	image: ${REPO}:python-${PYTHON}-wheel-windows-vs2019-vcpkg-${VCPKG}-${PYTHON_WHEEL_WINDOWS_IMAGE_REVISION}
	build:
	args:
	base: ${REPO}:python-wheel-windows-vs2019-base-vcpkg-${VCPKG}-${PYTHON_WHEEL_WINDOWS_IMAGE_REVISION}
	python: ${PYTHON}
	context: .
	dockerfile: ci/docker/python-wheel-windows-vs2019.dockerfile
	# This should make the pushed images reusable, but the image gets rebuilt.
	# Uncomment if no local cache is available.
	# cache_from:
	# - abrarov/msvc-2019:2.11.0
	# - ${REPO}:python-${PYTHON}-wheel-windows-vs2019-vcpkg-${VCPKG}-${PYTHON_WHEEL_WINDOWS_IMAGE_REVISION}
	volumes: *python-wheel-windows-vs2019-volumes
	command: arrow\\ci\\scripts\\python_wheel_windows_build.bat

Which sets the infrastructure and uses python_wheel_windows_build.bat to build the wheel.

[protokoul]

I have Windows and running docker desktop in it. I have the tried the following commands separately to check if I can run python-wheel-windows-vs2019 image but could not get it to work.

archery docker run python-wheel-windows-vs2019
archery docker run --no-cache python-wheel-windows-vs2019

Log shared below for the second command.

E:\github_forks\arrow-fork>archery docker run --no-cache python-wheel-windows-vs2019
[+] Pulling 1/1
 ! python-wheel-windows-vs2019-base Warning manifest for apache/arrow-dev:python-wheel-windows-vs...                                           5.0s
WARNING: Some service image(s) must be built from source by running:
    docker compose build python-wheel-windows-vs2019-base
[+] Pulling 1/1
 ! python-wheel-windows-vs2019 Warning manifest for apache/arrow-dev:python-3.9-wheel-windows-vs2019-v...                                      3.6s
WARNING: Some service image(s) must be built from source by running:
    docker compose build python-wheel-windows-vs2019
[+] Building 596.0s (5/13)                                                                                                     docker:desktop-linux
 => [python-wheel-windows-vs2019-base internal] load build definition from python-wheel-windows-vs2019-base.dockerfile                         0.0s
[+] Building 596.5s (5/13)                                                                                                     docker:desktop-linux
 => [python-wheel-windows-vs2019-base internal] load build definition from python-wheel-windows-vs2019-base.dockerfile                         0.0s
 => => transferring dockerfile: 3.44kB                                                                                                         0.0s
 => [python-wheel-windows-vs2019-base internal] load metadata for docker.io/abrarov/msvc-2019:2.11.0                                           3.1s
 => [python-wheel-windows-vs2019-base internal] load .dockerignore                                                                             0.0s
 => => transferring context: 1.78kB                                                                                                            0.0s
 => ERROR [python-wheel-windows-vs2019-base 1/9] FROM docker.io/abrarov/msvc-2019:2.11.0@sha256:164aabdb3f9dbef34bf558703de6b8936aa84e66553  592.3s
 => => resolve docker.io/abrarov/msvc-2019:2.11.0@sha256:164aabdb3f9dbef34bf558703de6b8936aa84e66553778192334cb3b8ba40a6e                      0.0s
 => => sha256:164aabdb3f9dbef34bf558703de6b8936aa84e66553778192334cb3b8ba40a6e 3.61kB / 3.61kB                                                 0.0s
 => => sha256:aa4f58cd6da1aaf1a0b44d443bd88e7fbe5b0a6f193995a1a61d6bd63990f314 672.54MB / 672.54MB                                           484.7s
 => => sha256:8e1b13b128117ce607a6063200c8e64fec84f3721dd3f653de194d890dc5adfe 7.28kB / 7.28kB                                                 0.0s
 => => sha256:4612f6d0b889cad0ed0292fae3a0b0c8a9e49aff6dea8eb049b2386d9b07986f 1.72GB / 1.72GB                                               458.0s
 => => sha256:769d8ca32f121d9cf6e507316bd89ca4a8883c69ab79307586d438417c09a56c 1.09kB / 1.09kB                                                 0.4s
 => => sha256:1de2eeb8c7149760430b5aa5273edf928eeab741767854a72c113ba71543071b 3.39kB / 3.39kB                                                 0.7s
 => => sha256:64b219704f84bf358c0ebb695b7244c8b0834a86f498c1870326009c564a9f30 1.10GB / 1.10GB                                                78.0s
 => => sha256:827099bae99f0034a43a461066f79307c725fa86848789631411a48977079d2c 1.12kB / 1.12kB                                                79.0s
 => => sha256:48387b0c01bd07f5fb0c37d1a3f1540bdb8833b16c4bd77630af7aa507227730 1.09kB / 1.09kB                                                79.3s
 => => sha256:88b8f096b457c000fd6bda2666584616c9abf20377c6437ac20712d33d6b3695 1.09kB / 1.09kB                                                79.6s
 => => sha256:2b67d29ad0b5fced82b44faab964319ce6fc1fd7f0a02155ffb23aa2b9b79c91 1.08kB / 1.08kB                                                80.0s
 => => sha256:302bd5f8d6b889b2e53eb8e23f77cd244059deb847a25a0853234cc117033be7 2.26kB / 2.26kB                                                80.3s
 => => sha256:2dbe5650ed9509d71dfbd4befb037c08dc8cd732d0ead29acfa556dbff1fff3a 2.79GB / 2.79GB                                               277.6s
 => => sha256:c2e46e68788722e963dc898656bb880083e2543234dde56fee2cfc7f96d5f03f 1.09kB / 1.09kB                                               280.0s
 => => sha256:1d79ed9a6e8196edbec96a27f4997d5202c9b575f4380eea9af8376cdfe2c75f 1.09kB / 1.09kB                                               280.3s
 => => sha256:7ea0ba46cf4f429c3fe3acd78c93781b79a1bab7acecd0b6ee1402c3b90a2cee 1.09kB / 1.09kB                                               280.8s
 => => extracting sha256:4612f6d0b889cad0ed0292fae3a0b0c8a9e49aff6dea8eb049b2386d9b07986f                                                    122.0s
 => => extracting sha256:aa4f58cd6da1aaf1a0b44d443bd88e7fbe5b0a6f193995a1a61d6bd63990f314                                                     10.8s
 => [python-wheel-windows-vs2019-base internal] load build context                                                                             2.0s
 => => transferring context: 338B                                                                                                              2.0s
------
 > [python-wheel-windows-vs2019-base 1/9] FROM docker.io/abrarov/msvc-2019:2.11.0@sha256:164aabdb3f9dbef34bf558703de6b8936aa84e66553778192334cb3b8ba[+] Building 0/1
 - Service python-wheel-windows-vs2019-base  Building                                                                                        597.5s
failed to solve: failed to register layer: link /Files/Program Files/common files/Microsoft Shared/Ink/en-US/micaut.dll.mui /Files/Program Files (x86)/common files/Microsoft Shared/ink/en-US/micaut.dll.mui: no such file or directory
Error: `docker compose --file=E:\github_forks\arrow-fork\docker-compose.yml build --no-cache --build-arg BUILDKIT_INLINE_CACHE=1 python-wheel-windows-vs2019-base` exited with a non-zero exit code 17, see the process log above.

There are warning in the logs indicating that some service images need to build from source by running docker compose build python-wheel-windows-vs2019-base and docker compose build python-wheel-windows-vs2019 so I tried docker compose build python-wheel-windows-vs2019-base first but that also did not work

E:\github_forks\arrow-fork>docker compose build python-wheel-windows-vs2019-base
[+] Building 542.8s (5/13)
 => [internal] load build definition from python-wheel-windows-vs2019-base.dockerfile                                                          0.2s
 => => transferring dockerfile: 3.44kB                                                                                                         0.1s
 => [internal] load .dockerignore                                                                                                              0.5s
 => => transferring context: 1.78kB                                                                                                            0.5s
 => [internal] load metadata for docker.io/abrarov/msvc-2019:2.11.0                                                                            5.6s
 => ERROR [1/9] FROM docker.io/abrarov/msvc-2019:2.11.0@sha256:164aabdb3f9dbef34bf558703de6b8936aa84e66553778192334cb3b8ba40a6e              536.9s
 => => resolve docker.io/abrarov/msvc-2019:2.11.0@sha256:164aabdb3f9dbef34bf558703de6b8936aa84e66553778192334cb3b8ba40a6e                      0.0s
 => => sha256:4612f6d0b889cad0ed0292fae3a0b0c8a9e49aff6dea8eb049b2386d9b07986f 1.72GB / 1.72GB                                               470.5s
 => => sha256:aa4f58cd6da1aaf1a0b44d443bd88e7fbe5b0a6f193995a1a61d6bd63990f314 672.54MB / 672.54MB                                           333.7s
 => => sha256:164aabdb3f9dbef34bf558703de6b8936aa84e66553778192334cb3b8ba40a6e 3.61kB / 3.61kB                                                 0.0s
 => => sha256:8e1b13b128117ce607a6063200c8e64fec84f3721dd3f653de194d890dc5adfe 7.28kB / 7.28kB                                                 0.0s
 => => sha256:769d8ca32f121d9cf6e507316bd89ca4a8883c69ab79307586d438417c09a56c 1.09kB / 1.09kB                                                 0.4s
 => => sha256:1de2eeb8c7149760430b5aa5273edf928eeab741767854a72c113ba71543071b 3.39kB / 3.39kB                                                 0.7s
 => => sha256:64b219704f84bf358c0ebb695b7244c8b0834a86f498c1870326009c564a9f30 1.10GB / 1.10GB                                                77.1s
 => => sha256:827099bae99f0034a43a461066f79307c725fa86848789631411a48977079d2c 1.12kB / 1.12kB                                                78.0s
 => => sha256:48387b0c01bd07f5fb0c37d1a3f1540bdb8833b16c4bd77630af7aa507227730 1.09kB / 1.09kB                                                78.4s
 => => sha256:88b8f096b457c000fd6bda2666584616c9abf20377c6437ac20712d33d6b3695 1.09kB / 1.09kB                                                78.7s
 => => sha256:2b67d29ad0b5fced82b44faab964319ce6fc1fd7f0a02155ffb23aa2b9b79c91 1.08kB / 1.08kB                                                79.0s
 => => sha256:302bd5f8d6b889b2e53eb8e23f77cd244059deb847a25a0853234cc117033be7 2.26kB / 2.26kB                                                79.3s
 => => sha256:2dbe5650ed9509d71dfbd4befb037c08dc8cd732d0ead29acfa556dbff1fff3a 2.79GB / 2.79GB                                               269.6s
 => => sha256:c2e46e68788722e963dc898656bb880083e2543234dde56fee2cfc7f96d5f03f 1.09kB / 1.09kB                                               271.6s
 => => sha256:1d79ed9a6e8196edbec96a27f4997d5202c9b575f4380eea9af8376cdfe2c75f 1.09kB / 1.09kB                                               272.0s
 => => sha256:7ea0ba46cf4f429c3fe3acd78c93781b79a1bab7acecd0b6ee1402c3b90a2cee 1.09kB / 1.09kB                                               272.3s
 => => extracting sha256:4612f6d0b889cad0ed0292fae3a0b0c8a9e49aff6dea8eb049b2386d9b07986f                                                     49.9s
 => => extracting sha256:aa4f58cd6da1aaf1a0b44d443bd88e7fbe5b0a6f193995a1a61d6bd63990f314                                                     15.2s
 => [internal] load build context                                                                                                             11.4s
 => => transferring context: 11.83kB                                                                                                          11.4s
------
 > [1/9] FROM docker.io/abrarov/msvc-2019:2.11.0@sha256:164aabdb3f9dbef34bf558703de6b8936aa84e66553778192334cb3b8ba40a6e:
------
failed to solve: failed to register layer: Error processing tar file(exit status 1): link /Files/Program Files/common files/Microsoft Shared/Ink/en-US/micaut.dll.mui /Files/Program Files (x86)/common files/Microsoft Shared/ink/en-US/micaut.dll.mui: no such file or directory

[raulcd]

I think this is an issue related to Windows containers, I can see some people facing the same issue and a potential fix here:
Ultimaker/cura-build-environment#90 (comment)
I don't have a Windows machine available so it's quite difficult for me to reproduce, hopefully this gives some possible ideas.

[protokoul]

Thanks for sharing this link. It seems that I will need Windows 11 Pro or Enterprise in order to run Windows containers. Let me see if I can get access to a machine with Windows Pro or Enterprise.

May I know how would you test windows related changes in the absence of a windows machine? Do you do it directly in CI using crossbow?

[raulcd]

Do you do it directly in CI using crossbow?

Unfortunately yes.

[protokoul]

Can I also try that because I will not be able to get a Windows 11 Pro or Enterprise to run Windows containers?

[raulcd]

You will not have permissions to trigger jobs via crossbow. If you have a PR up a committer can help triggering the jobs but this might complicate your development cycle as it will require manual intervention from someone else. I am happy to trigger the jobs but I might take some time to respond.