Failing to compile wheel for windows

[chmike]

Disclaimer: I’m a unix programmer with not much experience in windows programming

I have a very simple hello world program and I’m trying to build wheels using github actions.

The source code I’m compiling is as follow:

#ifdef _MSC_VER
#define _CRT_SECURE_NO_WARNINGS 1
#endif

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

// hello return a heap allocated string containing the name appended 
// to "hello " and followed by "!".
char *hello(char *name) {
    char *buf = malloc(7+strlen(name));
    sprintf(buf, "hello %s!", name);
    return buf;
}

The compilation output is the following:

  + pip wheel D:\a\hello_chmike\hello_chmike -w C:\Users\RUNNER~1\AppData\Local\Temp\cibuildwheele2yb0nd8\built_wheel --no-deps
  Processing d:\a\hello_chmike\hello_chmike
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Building wheels for collected packages: hello-chmike
    Building wheel for hello-chmike (PEP 517): started
    Building wheel for hello-chmike (PEP 517): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: 'c:\cibw\python\pythonx86.3.6.8\tools\python.exe' 'c:\cibw\python\pythonx86.3.6.8\tools\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\RUNNER~1\AppData\Local\Temp\tmp535cr0qk'
         cwd: C:\Users\RUNNER~1\AppData\Local\Temp\pip-req-build-ia9b5dcu
    Complete output (18 lines):
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build\lib.win32-3.6
    creating build\lib.win32-3.6\hello
    copying hello\__init__.py -> build\lib.win32-3.6\hello
    running build_ext
    building 'hello.ext' extension
    creating build\temp.win32-3.6
    creating build\temp.win32-3.6\Release
    creating build\temp.win32-3.6\Release\src
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\bin\HostX86\x86\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Isrc -Ic:\cibw\python\pythonx86.3.6.8\tools\include -Ic:\cibw\python\pythonx86.3.6.8\tools\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" /Tcsrc/hello.c /Fobuild\temp.win32-3.6\Release\src/hello.obj
  Failed to build hello-chmike
    hello.c
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\bin\HostX86\x86\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:c:\cibw\python\pythonx86.3.6.8\tools\libs /LIBPATH:c:\cibw\python\pythonx86.3.6.8\tools\PCbuild\win32 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\ATLMFC\lib\x86" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\lib\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x86" /EXPORT:PyInit_ext build\temp.win32-3.6\Release\src/hello.obj /OUT:build\lib.win32-3.6\hello\ext.cp36-win32.pyd /IMPLIB:build\temp.win32-3.6\Release\src\ext.cp36-win32.lib
    LINK : error LNK2001: unresolved external symbol PyInit_ext
    build\temp.win32-3.6\Release\src\ext.cp36-win32.lib : fatal error LNK1120: 1 unresolved externals
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.28.29333\\bin\\HostX86\\x86\\link.exe' failed with exit status 1120
    ----------------------------------------
    ERROR: Failed building wheel for hello-chmike
  ERROR: Failed to build one or more wheels

I don’t know what this PyInit_ext is. I didn’t expect I would have to do anything special about my c code. When I compile my code manually into a shared library on linux I have no problems.

[YannickJadoul]

@chmike, I'm not exactly sure why it's trying to link against PyInit_ext, but you're not compiling a Python C extension. Why exactly would you do that with cibuildwheel?

PyInit_ext is the entry point of a Python C extension called ext, btw. So I'm imagining something is checking that that symbol is present, somehow?

[chmike]

Thank you very much for responding. I'm not sure I understood your question and comment.

This is just a hello world c library that is called by a python wrapper using ctypes. When compiled locally it works and I successfully ran the test using pytest. I used pip install -e .. So the code is OK for linux, and as far as I understand it is C library extension to python but not using python. The purpose is to have an example I can use as reference implementation.

I need the wheels for easy install on windows. cibuildwheel looked like the perfect tool to do that.

[YannickJadoul]

Oh, right. I had overlooked the easy way of doing things!

So, how does your setup.py look? Judging from the output, it still contains a setuptools.Extension or so? So I think Python does think you're trying to create a C extension with a PyInit_modulename entry point (cfr. https://docs.python.org/3/extending/building.html).

[henryiii]

I do know Windows links extensions against python39.lib, while Linux / macOS do not. I think you need to set this up without making it an Extension.

[chmike]

The difference between unix or macOS and windows is strange. Shouldn't building wheel behave in the same way on all OS so that code is portable ?

My C library is totally independent of python and doesn't include python.h.

The module is here: https://github.com/chmike/hello_chmike

[mayeut]

Since you're building a setuptools.Extension , setuptools expect a PyInit_ext symbol to be present. Given how things are done by default (all symbols exported on unix/macOS, no symbols exported on Windows), the link command generated by setuptools forces the expected symbol to be exported on Windows with the switch /EXPORT:PyInit_ext.

[chmike]

Thank you for the explanation. All tutorials and examples I saw are using Extension.

Is there an easy solution with setuptools or equivalent to build the non-extension external C library that I can call with python by using the ctypes wrapper ?

[chmike]

After desperately trying to make my small example using ctypes compile and run on Windows, I give up. Defining a dll exported dummy variable named PyInt_ext solved the compilation error, but the test fails when I try to load the library libc.so.6 which can’t be found on Windows.

The conclusion is that to get a portable precompiled module, I do need to create a real python extension that wraps my python C agnostic library. I wish I had this information when I started.

[chmike]

Thank you very much for your interest in my problem.

Since I use ctypes which explicitly binds to the exported functions I want, the symbol PyInit_ext is not used. So it can be anything and the compiler stops complaining. Of course one shouldn't use it.

I needed libc.so.6 because I need the function free(). The free() is needed because my C library returns a heap allocated C string. In the wrapper I copy the string into a python object and then need to free the C string. This is required to avoid a memory leak. I have no idea how I should do that on windows.

Some people have suggested to add a free function in my library that then calls free(). That would be a way to avoid the need to load the libc.so.6 library.

[YannickJadoul]

I needed libc.so.6 because I need the function free(). The free() is needed because my C library returns a heap allocated C string. In the wrapper I copy the string into a python object and then need to free the C string. This is required to avoid a memory leak. I have no idea how I should do that on windows.

Yeah, but why is setuptools/distutils linking to libc.so.6? That's a Unix library. Windows has its own MSVCRT, no? I don't get why things don't get linked against that :-/

[YannickJadoul]

Some people have suggested to add a free function in my library that then calls free(). That would be a way to avoid the need to load the libc.so.6 library.

That might not be a bad idea anyway, by the way. I thought it's good practice to ensure you free things in the same library as you allocate them?

[chmike]

why is setuptools/distutils linking to libc.so.6?

It's not setuptool. It's me in the python code wrapping the library calls with ctypes. I try to load that library which does not exist by default on Windows. Setuptools/distutils is doing its job well in this case.

[Czaki]

So create own_free function in your library.
Your code will also fail in some of unx like system (for example alpine linux has musl instead of glibc)

Allow compiler do its job.

[chmike]

Thanks for the suggestion. Why would using `ctypes` be a preferred path compared to writing a real python extension ? Is it because it avoids dependencies to python versions ? Bien cordialement, Ch.Meessen Le 20 févr. 2021 à 12:24 +0100, Grzegorz Bokota <[email protected]>, a écrit :

…

So create own_free function in your library. Your code will also fail in some of unx like system (for example alpine linux has musl instead of glibc) Allow compiler do its job. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[YannickJadoul]

Right. Possibly that would get fixed in the context of #569

[henryiii]

You would need to manually rename the wheel, I believe (with or without cibuildwheel). Otherwise, sure.

[Czaki]

You would need to manually rename the wheel, I believe (with or without cibuildwheel). Otherwise, sure.

why? If there is no Extension entry in setup.py then should be no python version in name of a wheel.

[henryiii]

But would the platform then be in the name of the wheel? You need that.

[Czaki]

I see. I'm not sure.