Remote execution fails during Find interpreter for constraints: CPython==3.11.*
#21747
Comments
|
I feel like the issue is somewhere in this for loop, but only during execution, not when writing the script to disk. |
|
I'd investigate why the directory is empty. The issue is probably with one of the I don't have access to a remote execution environment. If you can run Pants from sources, then apply this patch and see what comes: |
|
You can run Pants from sources by cloning the Pants repository somewhere else on your system. Apply the patch above with |
|
Ok, I finally got somewhere. It looks like when trying to make the 08:46:36.52 [DEBUG] make_wrapper_for_append_only_caches: cache_name: CacheName("pex_root"); path: RelativePath(".cache/pex_root")
08:46:36.52 [DEBUG] make_wrapper_for_append_only_caches: mkdir parent: ".cache"
08:46:36.52 [DEBUG] make_wrapper_for_append_only_caches: cache_name: CacheName("python_build_standalone"); path: RelativePath(".python-build-standalone")
08:46:36.52 [DEBUG] make_wrapper_for_append_only_caches: mkdir parent: ""
08:46:36.53 [DEBUG] Completed: acquire_command_runner_slot
08:46:36.53 [DEBUG] Running Find interpreter for constraints: CPython==3.11.* under semaphore with concurrency id: 2, and concurrency: 1
08:46:36.53 [DEBUG] make_wrapper_for_append_only_caches: cache_name: CacheName("pex_root"); path: RelativePath(".cache/pex_root")
08:46:36.53 [DEBUG] make_wrapper_for_append_only_caches: mkdir parent: ".cache"
08:46:36.53 [DEBUG] make_wrapper_for_append_only_caches: cache_name: CacheName("python_build_standalone"); path: RelativePath(".python-build-standalone")
08:46:36.53 [DEBUG] make_wrapper_for_append_only_caches: mkdir parent: "" |
|
I think it is any path without a Output: The fix should be easy enough. We should just skip emitting a Try this patch: |
|
That did fix the issue, but now I'm running into BuildGrid issues, as it is responding with Rather than continue down this rabbit hole, I'm gonna try another REAPI. |
|
I tried again using BuildFarm instead of BuildGrid, and something I find odd is, the #!/bin/sh
/bin/mkdir -p '/tmp/named-caches/pex_root'
/bin/mkdir -p '.cache'
/bin/ln -s '/tmp/named-caches/pex_root' '.cache/pex_root'
/bin/mkdir -p '/tmp/named-caches/python_build_standalone'
/bin/ln -s '/tmp/named-caches/python_build_standalone' '.python-build-standalone'
exec "$@"What's populating the append-only caches? The absolute path supplied I also recognize that I could be completely missing something, but there doesn't appear to be anything in the debug logs that would populate the named caches with the correct data. |
The named caches are there for the build action to populate. The wrapper script exists solely to setup the correct directories so the build action can assume the directories exist. In local execution, the Pants engine would have done this directly without need for a wrapper script before it executed the build action. |
Then how would the execution be able to reference the venv-ed python binary without anything to bootstrap it? |
|
Pants use Pex to bootstrap venv's. You will note one of the named caches is for pex ( |
Understood, but how can the remote server execute a reference on my local machine? |
It can't. You are seeing the experimental nature of append-only / named caches in remote execution. Taking a remotely valid result and trying to run it locally with hard-coded absolute paths is a problem. (I.e., "non-hermetic" build actions) There could be a number of these problems to discover. Which is to say |
Ah, fair enough. I might mess around with it a bit more to see if I can make some progress. |
|
I switched to BuildFarm and got significant progress. I made a dumb and was running remote executions with my The patch you provided works. I can successfully remote execute. |
|
#21753 has the empty path fix. |
…#21753) As reported in #21747, the remote execution wrapper used to set up append-only cache directories was trying to create an empty path as a directory. This occurred because `Path::parent` returns an empty string (`Some("")`) if the path was not absolute (i.e., the "root" was not `""` and not `"/"`). Solution: Check for that condition and do not emit the `mkdir` in that case.
… (Cherry-pick of #21753) (#21760) As reported in #21747, the remote execution wrapper used to set up append-only cache directories was trying to create an empty path as a directory. This occurred because `Path::parent` returns an empty string (`Some("")`) if the path was not absolute (i.e., the "root" was not `""` and not `"/"`). Solution: Check for that condition and do not emit the `mkdir` in that case. Co-authored-by: Tom Dyas <[email protected]>
Describe the bug
The remote BuildGrid server is configured with a remote path of
/home/ssm-user/cachesand it has write access. Thepants.tomlhas the configurationremote_execution_append_only_caches_base_path = "/home/ssm-user/caches", which seems to work, as it createdpex_rootandpython_build_standalonedirectories in it. However, when I attempt to run a remote buildpants package ..., I get the following output:Pants version
2.23.0
OS
Local:
Linux GILR-147S2F3 5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024 x86_64 x86_64 x86_64 GNU/LinuxRemote:
Linux ip-10-192-13-117.ec2.internal 6.1.115-126.197.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 5 17:36:57 UTC 2024 x86_64 x86_64 x86_64 GNU/LinuxAdditional info
Everything works locally. Even the remote caching is producing cache hits. The remote execution seems to be missing something, but I'm unsure as to what that is.
The text was updated successfully, but these errors were encountered: