Option to avoid download python exe from https://github.com/indygreg/python-build-standalone #154

da-liii · 2023-04-05T17:49:32Z

For example, in Github Action, we'd prefer to install the python needed by Pants itself by actions/setup-python.

The text was updated successfully, but these errors were encountered:

jsirois · 2023-04-13T20:17:14Z

@darcy-shen that's a no-go. The whole point of scie-pants is that it ships with a Python interpreter you, as the consumer, are unaware of. It is absolutely not intended to work with an external interpreter. Do you actually care about the interpreter it uses, or just that it doesn't download the interpreter dynamically? If its the latter, that can be accomodated. Right now we just ship a "thin" scie that does not include the Python interpreter - to save space - but downloads it as needed instead. We can also ship "fat" scies that include the interpreter. That said, downloads will still happen (of wheels) when a new Pants version is installed.

I guess what I really need here is more explanation of why you want to avoid https://github.com/indygreg/python-build-standalone - due to the technology, due to the runtime use of the network, or some other reason?

da-liii · 2023-04-15T16:33:05Z

due to the technology

I have found a better way to install Python, see pantsbuild/pants#17962

due to the runtime use of the network

Yes, it can be unstable for me. The network is not guaranteed to be stable.

da-liii · 2023-04-15T16:34:07Z

We can also ship "fat" scies that include the interpreter.

That's a great idea.

Instead of using raw `scie-jump` and a hand-crafted JSON lift manifest, use `science` and a TOML lift mabifest to build `scie-pants`. The `science` tool and its more friendly interface are the present and future of end-user scie construction. We convert here and gain a few things: + A manifest format we can comment. + Abstraction over platforms and PBS details. + Support for embedded build provenanace information for more comprehensive black box debugability. + A clean path to publishing both "thin" and "fat" binaries and solving pantsbuild#154.

Instead of using raw `scie-jump` and a hand-crafted JSON lift manifest, use `science` and a TOML lift manifest to build `scie-pants`. The `science` tool and its more friendly interface are the present and future of end-user scie construction. We convert here and gain a few things: + A manifest format we can comment. + Abstraction over platforms and PBS details. + Support for embedded build provenanace information for more comprehensive black box debugability. + A clean path to publishing both "thin" and "fat" binaries, solving #154.

jsirois · 2023-05-27T18:16:10Z

With scie-pants now converted to science for building its scie, creating fat scies for release is as easy as adding --invert-lazy cpython39 --app-name scie-pants-fat to the science command here:

scie-pants/package/src/scie_pants.rs

Lines 37 to 61 in 14ed784

    
           execute( 
        
               Command::new(&skinny_scie_tools.science) 
        
                   .args([ 
        
                       "lift", 
        
                       "--include-provenance", 
        
                       "--file", 
        
                       &format!( 
        
                           "scie-pants.bin={scie_pants_exe}", 
        
                           scie_pants_exe = path_as_str(scie_pants_exe)? 
        
                       ), 
        
                       "--file", 
        
                       &format!( 
        
                           "tools.pex={tools_pex}", 
        
                           tools_pex = path_as_str(tools_pex_file)? 
        
                       ), 
        
                       "build", 
        
                       "--dest-dir", 
        
                       path_as_str(&scie_pants_package_dir)?, 
        
                       "--use-platform-suffix", 
        
                       "--hash", 
        
                       "sha256", 
        
                       path_as_str(&scie_pants_manifest)?, 
        
                   ]) 
        
                   .current_dir(&build_context.workspace_root), 
        
           )?;

The question that needs to be considered is whether to make the fat scie fully fat (Also include --invert-lazy cpython38). Although this question is probably easy to answer "no" to since 3.8 is only needed for Pants < 2.5 which is quite old at this point, it gets harder to answer when Pants switches to using CPython 3.11. At that point there will be very recent versions of Pants that need 3.9; so the fat variant including both cpython39 and cpython311 distributions (but still probably not cpython38) may make more sense.

I'll leave all this for others to kvetch over and implement.

cognifloyd · 2024-08-23T01:57:50Z

I like to use termux on my android phone, but it is a quirky environment. Most apps require patching to support the prefixed/relocated location for libs and other lookup paths. I can't use scie-pants because the underlying crates can't find resolve.conf and related networking bits, so dns lookups fail.

Even if the networking bits were resolved, PBS does not have a build for termux (astral-sh/python-build-standalone#176 and astral-sh/python-build-standalone#87). However, I finally found a pre-built python 3.9 interpreter for termux, so I would love to make scie-pants just use that. Relevant links:

the build recipe
the built binary deb files:

To install these deb files I run pkg install tur-repo and pkg install python3.9 where pkg wraps apt.

kaos · 2024-08-27T05:44:32Z

I like to use termux on my android phone, but it is a quirky environment. Most apps require patching to support the prefixed/relocated location for libs and other lookup paths. I can't use scie-pants because the underlying crates can't find resolve.conf and related networking bits, so dns lookups fail.

Even if the networking bits were resolved, PBS does not have a build for termux (indygreg/python-build-standalone#176 and indygreg/python-build-standalone#87). However, I finally found a pre-built python 3.9 interpreter for termux, so I would love to make scie-pants just use that. Relevant links:

the build recipe

the built binary deb files:

python3.9_3.9.18_aarch64.deb

python3.9_3.9.18_arm.deb

python3.9_3.9.18_i686.deb

python3.9_3.9.18_x86_64.deb

To install these deb files I run pkg install tur-repo and pkg install python3.9 where pkg wraps apt.

@cognifloyd could PANTS_BOOTSTRAP_URLS in combination with file:/// urls work for this? https://github.com/pantsbuild/scie-pants?tab=readme-ov-file#firewall-support

cognifloyd · 2024-08-27T14:51:16Z

@cognifloyd could PANTS_BOOTSTRAP_URLS in combination with file:/// urls work for this? https://github.com/pantsbuild/scie-pants?tab=readme-ov-file#firewall-support

Sweet. That's a helpful knob I didn't realize existed. So, I can now bypass the networking bits, but I think termux has to patch locking in Python to work with Android's libc, so I run into this error with the vanilla PBS:

Downloading file:///data/data/com.termux/files/home/downloads/ptex/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tar.gz...
Error: Failed to establish atomic directory /data/data/com.termux/files/home/.cache/nce/f4a11f9bb68736c1c58003a35df7d459dab771227cb6777ec70921f83e0cb95d/locks/configure-97635d72a1cd54a03e580cfb08efc421bd3a0ed9078f43a25e4e739885f2e937. Population of work directory failed: Failed to launch boot binding: Failed to spawn "/data/data/com.termux/files/home/.cache/nce/7d19e1ecd6e582423f7c74a0c67491eaa982ce9d5c5f35f0e4289f83127abcb8/cpython-3.9.18+20240107-aarch64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9" ["/data/data/com.termux/files/home/.cache/nce/ad8ed7b8c9b0e05355f776cd229a3b2c18840e000dd0c610048c92c3a910d317/tools.pex", "configure-pants", "--ptex-path", "/data/data/com.termux/files/home/.cache/nce/b71bea2deda31242e8d5d8749d173f4fe166182e8ebbef3ef605da06320bf24a/ptex-linux-aarch64", "--pants-version", "2.20.3", "--pants-config", "", "--github-api-bearer-token", "", "--pants-bootstrap-urls", "/data/data/com.termux/files/home/pants_urls.json", "/data/data/com.termux/files/home/.cache/nce/f4a11f9bb68736c1c58003a35df7d459dab771227cb6777ec70921f83e0cb95d/bindings"]: No such file or directory (os error 2)

This is the python compile/configure option that I think PBS is missing:
https://github.com/termux-user-repository/tur/blob/master/tur%2Fpython3.9%2Fbuild.sh#L30

edit: no it is probably not that patch. I think the issue is that the python binary can't find libc as it is linked against glibc not the android libc. I can't even run the PBS python binary that scie unpacked.)

$ ldd /data/data/com.termux/files/home/.cache/nce/7d19e1ecd6e582423f7c74a0c67491eaa982ce9d5c5f35f0e4289f83127abcb8/cpython-3.9.18+20240107-aarch64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9
        libpython3.9.so.1.0 => not found
        libpthread.so.0 => not found
        libdl.so.2 => not found
        libutil.so.1 => not found
        libm.so.6 => not found
        librt.so.1 => not found
        libc.so.6 => not found
$ /data/data/com.termux/files/home/.cache/nce/7d19e1ecd6e582423f7c74a0c67491eaa982ce9d5c5f35f0e4289f83127abcb8/cpython-3.9.18+20240107-aarch64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9
bash: /data/data/com.termux/files/home/.cache/nce/7d19e1ecd6e582423f7c74a0c67491eaa982ce9d5c5f35f0e4289f83127abcb8/cpython-3.9.18+20240107-aarch64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9: cannot execute: required file not found

Vs the termux build:

$ ldd $(which python3.9)
        libandroid-support.so => /data/data/com.termux/files/usr/lib/libandroid-support.so
        libpython3.9.so.1.0 => /data/data/com.termux/files/usr/lib/libpython3.9.so.1.0
        libc.so => /system/lib64/libc.so
        ld-android.so => /system/lib64/ld-android.so
        libdl.so => /system/lib64/libdl.so
        libm.so => /system/lib64/libm.so
$ python3.9
Python 3.9.18 (main, Sep  8 2023, 11:55:57)
[Clang 14.0.7 (https://android.googlesource.com/toolchain/llvm-project 4c603efb on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

kaos · 2024-08-30T04:44:06Z

You point it at a .tar.gz, doesn't have to be a archive containing the PBS dist, right? What if you package up your termux build and name that like the PBS one instead? (very hackish, indeed, but you are running this on your phone too... :P )

cognifloyd · 2024-08-30T12:57:59Z

You point it at a .tar.gz, doesn't have to be a archive containing the PBS dist, right? What if you package up your termux build and name that like the PBS one instead? (very hackish, indeed, but you are running this on your phone too... :P )

SCIE checks a hash of the file, so that won't work. 😞

scie-pants/README.md

Lines 96 to 99 in d336b8f

    
           The embedded artifact references also contain expected hashes of the downloaded content. Your 
        
           re-directed URLs must provide the same content as the canonical URLs; if the hashes of downloaded 
        
           files do not match those recorded in `scie-pants`, install will fail fast and let you know about 
        
           the hash mismatch.

kaos · 2024-08-30T14:36:05Z

You point it at a .tar.gz, doesn't have to be a archive containing the PBS dist, right? What if you package up your termux build and name that like the PBS one instead? (very hackish, indeed, but you are running this on your phone too... :P )

SCIE checks a hash of the file, so that won't work. 😞

scie-pants/README.md

Lines 96 to 99 in d336b8f

The embedded artifact references also contain expected hashes of the downloaded content. Your

re-directed URLs must provide the same content as the canonical URLs; if the hashes of downloaded

files do not match those recorded in `scie-pants`, install will fail fast and let you know about

the hash mismatch.

Doh. Forgot about that. Lol- Well, it sure is being secure at least. 😅

jsirois self-assigned this Apr 13, 2023

jsirois added the feature request Denotes a request for a new scie-pants feature. label Apr 26, 2023

jsirois mentioned this issue May 26, 2023

Build scie-pants using science. #191

Merged

jsirois added in progress Denotes an assigned issue is being actively worked on. and removed in progress Denotes an assigned issue is being actively worked on. labels May 27, 2023

jsirois removed their assignment May 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Option to avoid download python exe from https://github.com/indygreg/python-build-standalone #154

Option to avoid download python exe from https://github.com/indygreg/python-build-standalone #154

da-liii commented Apr 5, 2023

jsirois commented Apr 13, 2023

da-liii commented Apr 15, 2023

da-liii commented Apr 15, 2023

jsirois commented May 27, 2023 •

edited

Loading

cognifloyd commented Aug 23, 2024 •

edited

Loading

kaos commented Aug 27, 2024

cognifloyd commented Aug 27, 2024 •

edited

Loading

kaos commented Aug 30, 2024

cognifloyd commented Aug 30, 2024

kaos commented Aug 30, 2024

Option to avoid download python exe from https://github.com/indygreg/python-build-standalone #154

Option to avoid download python exe from https://github.com/indygreg/python-build-standalone #154

Comments

da-liii commented Apr 5, 2023

jsirois commented Apr 13, 2023

da-liii commented Apr 15, 2023

da-liii commented Apr 15, 2023

jsirois commented May 27, 2023 • edited Loading

cognifloyd commented Aug 23, 2024 • edited Loading

kaos commented Aug 27, 2024

cognifloyd commented Aug 27, 2024 • edited Loading

kaos commented Aug 30, 2024

cognifloyd commented Aug 30, 2024

kaos commented Aug 30, 2024

jsirois commented May 27, 2023 •

edited

Loading

cognifloyd commented Aug 23, 2024 •

edited

Loading

cognifloyd commented Aug 27, 2024 •

edited

Loading