Multi-stage build for Linux AArch64

[odidev]

Closes PyAV-Org/PyAV#708

[jlaine]

This is awesome, let's see how it goes!

[jlaine]

The Python script already invokes yum, do we need it here too?

[odidev]

Yum requires liblzma.so which is already present in the system. We are building a different version of liblzma.so for our own purpose and adding its path to LD_LIBRARY_PATH. If we allow script to execute yum; yum will detect our built liblzma.so which is incompatible causing installation failure.

To avoid this scenario we need to execute yum before adding liblzma.so's path to LD_LIBRARY_PATH.

[jlaine]

I don't think you need to touch LD_LIBRARY_PATH at all for stage1 and stage2, I think it's only actually needed to test the wheel which is built at the very end.

[odidev]

Please take a look at below code:

Package(
            name="gnutls",
            requires=["nettle", "unistring", "zlib"],
            source_url="https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.16.tar.xz",
            build_arguments=[
                "--disable-cxx",
                "--disable-doc",
                "--disable-libdane",
                "--disable-nls",
                "--disable-tests",
                "--disable-tools",
                "--with-included-libtasn1",
                "--without-p11-kit",
            ],
        ),

gnutls requires nettle, unistring and zlib. And these are being install on /tmp/vendor. So, to make it visible to build tool and use as a dependency we need to add /tmp/vendor to LD_LIBRARY_PATH. Please correct me if I am wrong.

However, I will do a side experiment on your suggestion and share you. :)

[jlaine]

LD_LIBRARY_PATH is not used when linking (we already handle this by setting LDFLAGS), it's used when a executable loads a shared library. The only case I can think of is when at the very end we test a wheel which is linked against ffmpeg (and transitively all its dependencies).

[odidev]

Removing LD_LIBRARY_PATH causes below failure-
Commit: odidev@4706d0d
Logs: https://github.com/odidev/pyav-ffmpeg/runs/5289695969?check_suite_focus=true#step:6:10805

[odidev]

Adding LDFLAGS should work but not sure why it is failing in above experiment.

[odidev]

It is failing while running configure script and I think LDFLAGS will not help here.

[jlaine]

This is very weird, but I can definitely reproduce it, my test build failed in the same way. I'll try and debug it. This really surprises me because the test in the gnutls configure script uses AC_CHECK_FUNCS which only relies on the ability to link a program, not execute it.

[odidev]

@jlaine can you please approve the workflow?

[jlaine]

Watch out, I see you're saving / restoring only /tmp/vendor. However following some refactoring there are two directories you want to save / restore:

• /tmp/vendor => these are the files for the "target"
• /tmp/vendor.builder => these are the files for the "builder". This is where you will find things like nasm and gperf (all packages with for_builder=True).

[jlaine]

I've launched my own experiment in #38

[jlaine]

Regarding LD_LIBRARY_PATH I've now remembered why I touch it at all: when repairing the wheels to incorporate external libraries, auditwheel uses LD_LIBRARY_PATH to determine its search path, see:

https://github.com/pypa/auditwheel/blob/f3c7691f258de76c11f679ab85826b730c658c14/src/auditwheel/lddtree.py#L187

I've opened an issue to see whether the auditwheel maintainers are open to adding an explicit option to pass additional paths:

pypa/auditwheel#373