SEACAS (Spack) hidden blas/lapack dependency #307

jjellio · 2022-05-27T00:39:56Z

I'm not sure where to report this - it's probably a spack issue - but it's manifesting with SEACAS' install.

Effectively, a package seacas is using is depending on lapack/blas to find _idmax_ and _idmin_. I suspect metis or parmetis. The error likely shows up, because I've built those packages statically, so they aren't able to silently RPATH in the required lib. Add to it, if built inside trilinos, lapack/blas are typically on the link lines anyway (so this could be an actual seacas or zoltan dependency? a quick grep says no).

The spack line is:

spack -d --verbose install   seacas ~matio+metis~x11 %[email protected] arch=cray-rhel8-zen3  \
  ^ cgns~base_scope~int64~ipo~legacy~mem_debug~fortran+hdf5+mpi+scoping+static+shared %[email protected] arch=cray-rhel8-zen3  \
  ^ netcdf-c ~hdf4~jna~dap +mpi+parallel-netcdf+shared %[email protected] arch=cray-rhel8-zen3 \
  ^ parallel-netcdf ~cxx~burstbuffer+fortran~shared %[email protected] arch=cray-rhel8-zen3 \
  ^ [email protected] ~cxx~threadsafe~java+fortran+hl+mpi~szip~shared %[email protected] arch=cray-rhel8-zen3 \
  ^ [email protected] ~gdb~ipo+int64+shared %[email protected] arch=cray-rhel8-zen3 \
  ^ metis@5: ~gdb+int64+real64+shared %[email protected] arch=cray-rhel8-zen3 \

The quick fix was to add depends_on('lapack') to the seacas package (when +metis). I'm not too familiar with SEACAS outside Trilinos - and I'm afriad posting this directly to spack will have it lost in the weeds rather fast.

==> [2022-05-26-17:08:33.132857] Error: ProcessError: Command exited with status 2:
    'make' '-j128'

45 errors found in build log:
     5334    [ 91%] Building Fortran object packages/seacas/applications/algebra/CMakeFiles/algebra.dir/ag_wnam.f.o
     5335    cd /tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-build-l23llax/packages/seacas/applications/algebra && /g/g20/jjellio/rzvernal/s
             pack/lib/spack/env/cce/ftn -DADDC_ -DBuild64 -I/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-build-l23llax -I/var/tmp/jjellio/sp
             ack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/applications/algebra -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-0
             3-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/suplib -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/s
             pack-src/packages/seacas/libraries/supes -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/supes
             /mem_mgr -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/supes/ext_lib -I/var/tmp/jjellio/spac
             k-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/supes/fre_fld -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-
             03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/exodus_for/include -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huv
             h7b4oj5tc2j/spack-src/packages/seacas/libraries/exodus_for/test -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages
             /seacas/libraries/exodus_for -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/exodus/include -I
             /tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-build-l23llax/packages/seacas/libraries/exodus -I/g/g20/jjellio/rzvernal/spack/opt
             /spack/cray-rhel8-zen3/cce-14.0.0/netcdf-c-4.8.1-gvk46omyrujsdskcgfjwe5qe4dbs6k53/include -I/g/g20/jjellio/rzvernal/spack/opt/spack/cray-rhel8-zen3/cce-14.0.0/hdf5-1.10.7-gi7x3
             nwecygku2lyrzmursorbqowv5rw/include -I/g/g20/jjellio/rzvernal/spack/opt/spack/cray-rhel8-zen3/cce-14.0.0/parallel-netcdf-1.12.2-woykzijolpwclh5d7zsenrgqwdroyilg/include -sdefau
             lt64 -em -J. -c /var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/applications/algebra/ag_wnam.f -o CMakeFil
             es/algebra.dir/ag_wnam.f.o
     5336    [ 91%] Building Fortran object packages/seacas/applications/algebra/CMakeFiles/algebra.dir/ag_wrstep.f.o
     5337    cd /tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-build-l23llax/packages/seacas/applications/algebra && /g/g20/jjellio/rzvernal/s
             pack/lib/spack/env/cce/ftn -DADDC_ -DBuild64 -I/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-build-l23llax -I/var/tmp/jjellio/sp
             ack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/applications/algebra -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-0
             3-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/suplib -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/s
             pack-src/packages/seacas/libraries/supes -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/supes
             /mem_mgr -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/supes/ext_lib -I/var/tmp/jjellio/spac
             k-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/supes/fre_fld -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-
             03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/exodus_for/include -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huv
             h7b4oj5tc2j/spack-src/packages/seacas/libraries/exodus_for/test -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages
             /seacas/libraries/exodus_for -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/exodus/include -I
             /tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-build-l23llax/packages/seacas/libraries/exodus -I/g/g20/jjellio/rzvernal/spack/opt
             /spack/cray-rhel8-zen3/cce-14.0.0/netcdf-c-4.8.1-gvk46omyrujsdskcgfjwe5qe4dbs6k53/include -I/g/g20/jjellio/rzvernal/spack/opt/spack/cray-rhel8-zen3/cce-14.0.0/hdf5-1.10.7-gi7x3
             nwecygku2lyrzmursorbqowv5rw/include -I/g/g20/jjellio/rzvernal/spack/opt/spack/cray-rhel8-zen3/cce-14.0.0/parallel-netcdf-1.12.2-woykzijolpwclh5d7zsenrgqwdroyilg/include -sdefau
             lt64 -em -J. -c /var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/applications/algebra/ag_wrstep.f -o CMakeF
             iles/algebra.dir/ag_wrstep.f.o
     5338    [ 91%] Building Fortran object packages/seacas/applications/gen3d/CMakeFiles/gen3d.dir/gen3d.f.o
     5339    cd /tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-build-l23llax/packages/seacas/applications/gen3d && /g/g20/jjellio/rzvernal/spa
             ck/lib/spack/env/cce/ftn -DADDC_ -DBuild64 -I/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-build-l23llax -I/var/tmp/jjellio/spac
             k-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/applications/gen3d -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04
             -l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/suplib -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack
             -src/packages/seacas/libraries/supes -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/supes/mem
             _mgr -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/supes/ext_lib -I/var/tmp/jjellio/spack-st
             age/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/supes/fre_fld -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-0
             4-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/exodus_for/include -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4
             oj5tc2j/spack-src/packages/seacas/libraries/exodus_for/test -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/sea
             cas/libraries/exodus_for -I/var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/libraries/exodus/include -I/tmp
             /jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-build-l23llax/packages/seacas/libraries/exodus -I/g/g20/jjellio/rzvernal/spack/opt/spa
             ck/cray-rhel8-zen3/cce-14.0.0/netcdf-c-4.8.1-gvk46omyrujsdskcgfjwe5qe4dbs6k53/include -I/g/g20/jjellio/rzvernal/spack/opt/spack/cray-rhel8-zen3/cce-14.0.0/hdf5-1.10.7-gi7x3nwec
             ygku2lyrzmursorbqowv5rw/include -I/g/g20/jjellio/rzvernal/spack/opt/spack/cray-rhel8-zen3/cce-14.0.0/parallel-netcdf-1.12.2-woykzijolpwclh5d7zsenrgqwdroyilg/include -sdefault64
              -em -J. -c /var/tmp/jjellio/spack-stage/spack-stage-seacas-2022-03-04-l23llaxi3daikyuqx7huvh7b4oj5tc2j/spack-src/packages/seacas/applications/gen3d/gen3d.f -o CMakeFiles/gen3d
             .dir/gen3d.f.o
  >> 5340    /opt/cray/pe/cce/14.0.0/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: ../../libraries/suplib/libsuplib.so.2.0: undefined reference to `_idmax_'
  >> 5341    /opt/cray/pe/cce/14.0.0/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: ../../libraries/suplib/libsuplib.so.2.0: undefined reference to `_idmin_'

The text was updated successfully, but these errors were encountered:

gsjaardema · 2022-05-27T14:02:13Z

Very strange -- It is indicating that _idmin_ and _idmax_ are being referenced from libsuplib.

/opt/cray/pe/cce/14.0.0/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: ../../libraries/suplib/libsuplib.so.2.0: undefined reference to `_idmax_'
/opt/cray/pe/cce/14.0.0/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: ../../libraries/suplib/libsuplib.so.2.0: undefined reference to `_idmin_'

These symbols don't appear in any of the files that are included in libsuplib as best I can tell... If they can be resolved from blas or lapack, then my guess is that they are being resolved incorrectly, so not sure if that is a good solution and may be instead hiding the problem...

I will see if I can figure out where this is coming from.... Somehow the fortran compiler seems to be doing this somehow...

gsjaardema · 2022-05-27T15:11:52Z

Is it possible to query the libsuplib.so.2.0 and see if the _idmax_ and _idmin_ are references to symbols or functions?
Maybe a nm -ls libsuplib.so.2.0 and see how _idmax_ and _idmin_ show up...

jjellio · 2022-05-27T16:34:13Z

Let me check and see. I did nm on libsci to vet that they were coming from the lapack/blas side. I strongly suspect it is coming from metis not declaring a blas dependence - but I wonder why I'm the first to run into this.

Let me get on that machine and dig a little. I do not think this is really a bug in SEACAS, but I also wasn't able to trigger it anywhere else =\

gsjaardema · 2022-05-27T16:56:01Z

SEACAS can be built without metis, so would be interesting to see if metis is truly where this is coming from. Confusing that it says the undefined references are from libsuplib.so.2.0 since that has no dependence at all on metis...

jjellio · 2022-05-27T17:06:22Z

Not metis/parmetis.

Weird, grepping all the libs I have from my spack build,

[jjellio cce-14.0.0]$ for f in $(ls -1  */lib/*.so); do v=$(nm -ls $f 2>/dev/null | grep -n _idmax_) && echo $f $v ; done
seacas-2022-03-04-auxrfqtbcafilt5zutiflmxpmw34ebfo/lib/libsuplib.so 251: U _idmax_

I wonder if cray fortran is doing some kind of optimization and promoting code to a libsci function?
Good question is if I disable metis/parmetis, if this error will hang around - libsci is definitely resolving the symbol.

gsjaardema · 2022-05-27T17:17:08Z

Yes, that was my only thought -- cray fortran optimizer changing a loop into a blas call... Seems like if it could do that it would also add the dependency, but maybe that is getting lost somewhere...

gsjaardema · 2022-09-12T23:02:11Z

@jjellio Is this still an issue, or did it get solved outside of here...

jjellio · 2022-09-13T18:55:11Z

It's not really solved. The grief happens if you unwrap Cray PE, and use CrayFTN directly w/optimizations turned on (O2/O3 seemed to do it). The result is, Cray Fortran transformed a region into a BLAS call, expecting Cray libsci to be on the command line (I'm guessing their PE simply falls over when the wrapper stuff is taken away - hence it doesn't know what to do). -- Edit add: It's spack that is yanking Cray's fortran compiler outside ftn - that's where where all the necessary info is getting lost.

It's really a SPACK thing - spack also removes libsci (explicitly) when you use Cray's wrappers - it only adds the libsci information back to the ENV if the package actually specifies a blas/lapack dependency... So, this is probably a spack specific issue (but it may arise again ... spack is trying to completely unwrap Cray, so maybe this will be become more prevalent.)

Maybe we just close this - and atleast someone can search github if pops up?

Edit add: using the Cray Fortran flag -hnopattern prevents this type of optimization and did provide one way to work around it.

gsjaardema · 2022-09-13T19:47:26Z

OK. Could also possibly add either a blas dependency or the -hnopattern if building under spack. Or just CRAY in general as the SEACAS fortran codes are not typically going to gain much from the transformations...

jjellio · 2022-09-13T20:16:25Z

My view - is there is zero wrong with your Spack package or SEACAS. I added a lapack dependency (and use libsci) in spack - but that just let's me work around it. If someone did CrayFTN via spack - but had blas marked something besides libsci, I'm guessing they would hit some archaic build error that makes little sense.

I seem to be the only one that's hit this, but SPACK is trying to unwrap Cray even more (so maybe this will happen more often). I guess you could

modify you spack package to have a lapack dependence
add a condition in your spack package to detect if the compiler is cray Fortran - then set -h nopattern
Do nothing, and see if this pops up again - we could loop in Todd Gamblin and see what he wants to do

gsjaardema · 2022-09-13T21:58:16Z

Yes, all good suggestions. Maybe best at this time is to just leave it as is and reopen if it happens again... Which it probably will...

I think (2) is probably the best solution for SEACAS since the added optimization to a blas routine probably saves a millisecond at most... I will add that to my "do when you have a few minutes to burn" queueu.

github-actions · 2022-11-18T02:15:51Z

Stale issue message

github-actions bot added the no-issue-activity label Nov 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SEACAS (Spack) hidden blas/lapack dependency #307

SEACAS (Spack) hidden blas/lapack dependency #307

jjellio commented May 27, 2022 •

edited

Loading

gsjaardema commented May 27, 2022

gsjaardema commented May 27, 2022 •

edited

Loading

jjellio commented May 27, 2022

gsjaardema commented May 27, 2022

jjellio commented May 27, 2022

gsjaardema commented May 27, 2022

gsjaardema commented Sep 12, 2022

jjellio commented Sep 13, 2022 •

edited

Loading

gsjaardema commented Sep 13, 2022

jjellio commented Sep 13, 2022

gsjaardema commented Sep 13, 2022

github-actions bot commented Nov 18, 2022

SEACAS (Spack) hidden blas/lapack dependency #307

SEACAS (Spack) hidden blas/lapack dependency #307

Comments

jjellio commented May 27, 2022 • edited Loading

gsjaardema commented May 27, 2022

gsjaardema commented May 27, 2022 • edited Loading

jjellio commented May 27, 2022

gsjaardema commented May 27, 2022

jjellio commented May 27, 2022

gsjaardema commented May 27, 2022

gsjaardema commented Sep 12, 2022

jjellio commented Sep 13, 2022 • edited Loading

gsjaardema commented Sep 13, 2022

jjellio commented Sep 13, 2022

gsjaardema commented Sep 13, 2022

github-actions bot commented Nov 18, 2022

jjellio commented May 27, 2022 •

edited

Loading

gsjaardema commented May 27, 2022 •

edited

Loading

jjellio commented Sep 13, 2022 •

edited

Loading