Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

musllinux python3.11 broken wheel #1559

Closed
MartinoMensio opened this issue Apr 4, 2023 · 6 comments · Fixed by #1560
Closed

musllinux python3.11 broken wheel #1559

MartinoMensio opened this issue Apr 4, 2023 · 6 comments · Fixed by #1560
Labels
bug Something isn't working

Comments

@MartinoMensio
Copy link

Bug Description

Hi,
Thanks for the great resource.
I am testing musllinux builds, and I am currently unable to generate working wheels for python3.11. When I use python3.10 or lower, the compilation works as expected.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/maturin_musl_fail/__init__.py", line 1, in <module>
    from .maturin_musl_fail import *
ModuleNotFoundError: No module named 'maturin_musl_fail.maturin_musl_fail'

I am not sure if this is a pyo3 isssue or a maturin issue, so please if this is not the correct place, just let me know.

Best,
Martino

Steps to Reproduce

  1. I created this minimal example repository: https://github.com/MartinoMensio/maturin_musl_fail with the code generated with maturin init
  2. I build the wheels with GitHub Actions, and they are generated successfully https://github.com/MartinoMensio/maturin_musl_fail/actions/runs/4608410699#artifacts
  3. I download the wheels and I run the following code:
# tested broken on platforms linux/amd64 and linux/386 with alpine (musllinux) 
docker run --rm -it --platform=linux/amd64 -v `pwd`/wheels:/wheels python:3.11-alpine sh
# then inside install wheel (everything ok)
pip install wheels/maturin_musl_fail-0.1.0-cp311-cp311-musllinux_1_1_x86_64.whl
# then test import
python -c "import maturin_musl_fail; print('import ok')"

The last step generates:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/maturin_musl_fail/__init__.py", line 1, in <module>
    from .maturin_musl_fail import *
ModuleNotFoundError: No module named 'maturin_musl_fail.maturin_musl_fail'

Backtrace

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/maturin_musl_fail/__init__.py", line 1, in <module>
    from .maturin_musl_fail import *
ModuleNotFoundError: No module named 'maturin_musl_fail.maturin_musl_fail'

Your operating system and version

MacOS Ventura 13.3

Your Python version (python --version)

python 3.11

Your Rust version (rustc --version)

rustc 1.68.2 (9eb3afe9e 2023-03-27)

Your PyO3 version

0.18.1

How did you install python? Did you use a virtualenv?

Buildtime:
github actions

Runtime generating error:
docker: python:3.11-alpine

Additional Info

No response

@MartinoMensio
Copy link
Author

After analysing the generated .so files and reading the guide here: https://pyo3.rs/main/building_and_distribution

I discovered that:

  1. the generated wheel has a .so file with name maturin_musl_fail.cpython-311-x86_64-linux-gnu.so
  2. running python -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))' generates .cpython-311-x86_64-linux-musl.so
  3. renaming the .so file to maturin_musl_fail.cpython-311-x86_64-linux-musl.so (changing gnu to musl) solves the issue and I can load the module.

Therefore I will now try to understand the source code of https://github.com/PyO3/maturin-action as it's probably what is setting the EXT_SUFFIX wrongly.

Any help understanding this is appreciated :)

Best,
Martino

@messense
Copy link
Member

messense commented Apr 5, 2023

See pypa/auditwheel#349, probably alpine updated SOABI on python 3.11, we may have to update in maturin too.

@messense messense transferred this issue from PyO3/pyo3 Apr 5, 2023
@MartinoMensio
Copy link
Author

Thank you very much @messense for looking into this!

Martino

bors bot added a commit that referenced this issue Apr 5, 2023
1560: Fix wrong `EXT_SUFFIX` when cross compiling musllinux wheels for Python 3.11 r=messense a=messense

Fixes #1559 

Co-authored-by: messense <[email protected]>
@bors bors bot closed this as completed in 3320f1a Apr 5, 2023
@messense
Copy link
Member

messense commented Apr 6, 2023

To test it, you'll need to use

- uses: PyO3/maturin-action@v1
  with:
    maturin-version: v1.0.0-beta.7

@MartinoMensio
Copy link
Author

Thank you very much! I tested it and it works perfectly!

@messense
Copy link
Member

messense commented Apr 6, 2023

@MartinoMensio Fix released in v0.14.17.

kraj added a commit to YoeDistro/poky that referenced this issue Oct 19, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

Signed-off-by: Khem Raj <[email protected]>
kraj added a commit to YoeDistro/poky that referenced this issue Oct 19, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

Signed-off-by: Khem Raj <[email protected]>
kraj added a commit to YoeDistro/poky that referenced this issue Oct 19, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

Signed-off-by: Khem Raj <[email protected]>
kraj added a commit to YoeDistro/poky that referenced this issue Oct 22, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

Signed-off-by: Khem Raj <[email protected]>
kraj added a commit to YoeDistro/poky that referenced this issue Oct 22, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

Signed-off-by: Khem Raj <[email protected]>
kraj added a commit to YoeDistro/poky that referenced this issue Oct 22, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

Signed-off-by: Khem Raj <[email protected]>
kraj added a commit to YoeDistro/poky that referenced this issue Oct 22, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

Signed-off-by: Khem Raj <[email protected]>
kraj added a commit to YoeDistro/poky that referenced this issue Oct 23, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

Signed-off-by: Khem Raj <[email protected]>
kraj added a commit to YoeDistro/poky that referenced this issue Oct 23, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

Signed-off-by: Khem Raj <[email protected]>
kraj added a commit to YoeDistro/poky that referenced this issue Oct 23, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

Signed-off-by: Khem Raj <[email protected]>
brainhoard-github pushed a commit to distro-core-curated-mirrors/poky-contrib that referenced this issue Oct 25, 2024
Currently, platform is based upon build_os detection which works ok
for native builds and where build_os and host_os are same i.e. glibc
based systems, but it fails for musl based systems to load correct
modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported
at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
meta-homeassistant/meta-homeassistant#89
pypa/auditwheel#349
PyO3/maturin#1559

(From OE-Core rev: 346cce305dcffa98bfabf8b14e5015a80b5d21ab)

Signed-off-by: Khem Raj <[email protected]>
Signed-off-by: Mathieu Dubois-Briand <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants