manywheel linux shared object issue

[telegraphic]

the generate-wheels.yml github action is missing something:

In [2]: import pygedm
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-2bed23ea31d2> in <module>
----> 1 import pygedm

~/mpy3/envs/p38/lib/python3.8/site-packages/pygedm/__init__.py in <module>
      6     __version__ = '0.0.1 - manual install'
      7
----> 8 from .pygedm import dm_to_dist, dist_to_dm, calculate_electron_density_lbr, calculate_electron_density_xyz, \
      9     calculate_halo_dm, generate_healpix_dm_map, convert_lbr_to_xyz

~/mpy3/envs/p38/lib/python3.8/site-packages/pygedm/pygedm.py in <module>
     22 """
     23 from . import ymw16_wrapper
---> 24 from . import ne2001_wrapper
     25 from . import yt2020
     26 from . import healpix_utils

~/mpy3/envs/p38/lib/python3.8/site-packages/pygedm/ne2001_wrapper.py in <module>
     19 from functools import wraps
     20 from astropy import units as u
---> 21 import ne21c
     22
     23 DATA_PATH = os.path.dirname(os.path.abspath(__file__))

ImportError: /home/dancpr/mpy3/envs/p38/lib/python3.8/site-packages/pygedm.libs/libf2c-e943422c.so.0.23: undefined symbol: MAIN__

In [3]: cd /home/dancpr/mpy3/envs/p38/lib/python3.8/site-packages/pygedm.libs/
/home/dancpr/mpy3/envs/p38/lib/python3.8/site-packages/pygedm.libs

That symbol is in the ne21c main.cpp, unsure what libf2c thinks its doing... This one might take a while to debug. For now I have deleted from PyPi the broken wheels.

[telegraphic]

Ok this one was a freaking nightmare, but I think it's now working!

The issue is a very weird one, in that libf2c.so "has a main object included which expects
a resolution of the symbol MAIN__ that is ostensibly provided by f2c when a fortran PROGRAM is compiled under it." (see http://www.verycomputer.com/183_d2c8799a512b77ba_1.htm).

I had previously gotten around this by adding a simple int MAIN__() { return 0;} in my C code as per suggestion, but this didn't seem to work for the so being built via the manywheel docker container.

The thing that worked was linking against libf2c.a, not the libf2c.so file. In the setup.py there is now -l:libf2c.a instead of -lf2c.

To make matters more complicated, CentOS used in the manylinux docker container does not include libf2c.a (ubuntu does). So I had to add a script to manually download and compile libf2c.a -- quite a pain.

It looks like the conda install -c conda-forge f2c gives a shared object that doesn't have the MAIN__ issue, so that path should be followed for use on MacOS.

[telegraphic]

It turns out that the manywheels linux build stopped complaining about MAIN__, but segfaulted when you tried to use the ne21c.so 😢

It also seems that auditwheel doesn't copy over the libf2c.a that is required.

I reverted to trying to use the libf2c.so again, and found that the -fvisibility=hidden flag added in BuildExt in setup.py is incompatible and is they underlying cause. I also found that the script did not pass along extra_compile_args as it was overwritten.

However, the wheel is still broken. I now find that even if I compile libf2c from source, or use the yum install f2c in the manylinux image, the resultant wheels fail with:

In [10]: ne21c.dist_to_dm(0,0,1)
invalid number: incomprehensible list input
apparent state: unit 11 named gal01.inp
last format: list io
lately reading direct formatted external IO
Aborted (core dumped)

I think there's something fundamentally wrong with libf2c on the CentOS image, and I am giving up for now. So no wheels until someone solves this...

[noREAVER]

Any solution to this? I run Fedora and have the same error.

[telegraphic]

Unfortunately not, I'd suggest trying the f2c in conda (conda install -c conda-forge f2c ) and installation by downloading then running pip install .