Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade ascent build system and use [email protected] on CI platforms #20

Merged
merged 36 commits into from
Oct 12, 2023

Conversation

nkoukpaizan
Copy link
Collaborator

This PR upgrades the Ascent build system to use a Spack submodule and install the dependencies in a common shared space.

This is not ready to merge yet (See #19).

@cameronrutherford
Copy link
Contributor

If ~python is what unblocks you for the moment on Ascent I think that is ok so long as we capture it in an issue and revisit it later.

Getting the functionality of a pipeline that can rebuilt is more important first so we can iterate quicker. If you want to try and work with GitLab syntax and write some pipelines for ORNL you can follow the example in .gitlab/pnnl, otherwise you can hand this off to me and I can take it from here.

@nkoukpaizan
Copy link
Collaborator Author

Okay, I'll take a look at the gitlab pipeline to build the modules.

@cameronrutherford
Copy link
Contributor

@rothpc and @nkoukpaizan not sure why but unzip isn't being found on the path in spack... This doesn't seem to be an issue for all packages, just cub?

Latest `MY_CLUSTER=ascent ./buildsystem/spack/install-compute.sh` :
/opt/ibm/spectrumcomputing/lsf/10.1.0.11/linux3.10-glibc2.17-ppc64le-csm/bin:/sw/sources/lsf-tools/2.0/ascent/bin:/opt/ibm/csm/bin:/opt/ibm/spectrumcomputing/lsf/10.1.0.11/linux3.10-glibc2.17-ppc64le-csm/etc:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibm/flightlog/bin:/opt/ibm/jsm/bin:/opt/puppetlabs/bin:/usr/lpp/mmfs/bin
/usr/bin/unzip
ascent
==> Created environment in /gpfs/wolf/proj-shared/csc359/cameron/exago/spack-env-ascent
==> You can activate this environment with:
==>   spack env activate /gpfs/wolf/proj-shared/csc359/cameron/exago/spack-env-ascent
spack configuration will be installed into /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install
spack mirror will be in /gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror
==> Configuring spec exago@=develop for development at path /gpfs/wolf/proj-shared/csc359/cameron/exago
spack-public [sb] file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror
local        [sb] file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror

Currently Loaded Modules:
  1) python/3.9-anaconda3



==> Concretized exago@develop+cuda cuda_arch=70
 -   kt57m33  exago@develop%[email protected]+cuda+hiop~ipo+ipopt+logging+mpi+python+raja~rocm build_system=cmake build_type=MinSizeRel cuda_arch=70 dev_path=/gpfs/wolf/proj-shared/csc359/cameron/exago generator=make arch=linux-rhel8-power9le
 -   fo2wqd7      ^[email protected]%[email protected]+cuda~ipo+openmp~rocm~tests build_system=cmake build_type=Release cuda_arch=70 generator=make arch=linux-rhel8-power9le
 -   a6k3ix4          ^[email protected]%[email protected] build_system=generic arch=linux-rhel8-power9le
 -   wqobtj5          ^[email protected]%[email protected] build_system=generic arch=linux-rhel8-power9le
[e]  xrvalu5      ^[email protected]%[email protected]~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rhel8-power9le
[e]  jtm4zyv      ^[email protected]%[email protected]~allow-unsupported-compilers~dev build_system=generic arch=linux-rhel8-power9le
 -   hrielr2      ^[email protected]%[email protected]~guile build_system=autotools arch=linux-rhel8-power9le
[+]  errn6vw          ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=linux-rhel8-power9le
 -   fsih7jo      ^hiop@develop%[email protected]+cuda~cusolver_lu~deepchecking+ginkgo~ipo~jsrun+kron+mpi+raja~rocm~shared+sparse build_system=cmake build_type=Debug cuda_arch=70 generator=make arch=linux-rhel8-power9le
 -   tujzksu          ^[email protected]%[email protected]+blas build_system=autotools arch=linux-rhel8-power9le
 -   4gj4k3k          ^[email protected]_experimental%[email protected]+cuda~develtools~full_optimizations~hwloc~ipo~mpi~oneapi+openmp~rocm+shared build_system=cmake build_type=Release cuda_arch=70 generator=make arch=linux-rhel8-power9le
 -   iw4c2as          ^[email protected]%[email protected]+cuda+fortran~ipo~rocm+shared build_system=cmake build_type=Release cuda_arch=70 generator=make arch=linux-rhel8-power9le
 -   5x3ymee          ^[email protected]%[email protected]~gdb~int64~ipo~real64+shared build_system=cmake build_type=Release generator=make patches=4991da9,93a7903,b1225da arch=linux-rhel8-power9le
 -   nurcezd          ^[email protected]%[email protected]~cuda~graphblas~openmp+pic build_system=generic arch=linux-rhel8-power9le
 -   k525wgg              ^[email protected]%[email protected]+cxx build_system=autotools libs=shared,static patches=69ad2e2 arch=linux-rhel8-power9le
 -   ird2vct                  ^[email protected]%[email protected] build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-rhel8-power9le
 -   v2ztfrg                  ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   fioxub5                  ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   zhqgx25              ^[email protected]%[email protected]+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rhel8-power9le
 -   yomz2ke                  ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   ouxdolh              ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=linux-rhel8-power9le
 -   hraabqt                  ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   5iadvvy                  ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   qv564ko                      ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools arch=linux-rhel8-power9le
 -   sgutdp7      ^[email protected]%[email protected]+coinhsl~debug~metis~mumps build_system=autotools arch=linux-rhel8-power9le
[e]  as2v3e6      ^[email protected]%[email protected]~bignuma~consistent_fpcsr~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-rhel8-power9le
 -   xpholqu      ^[email protected]%[email protected]~X~batch~cgns~complex~cuda~debug+double~exodusii~fftw+fortran~giflib~hdf5~hpddm~hwloc~hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind~metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse~superlu-dist~tetgen~trilinos~valgrind build_system=generic clanguage=C memalign=none arch=linux-rhel8-power9le
 -   u4bqjef          ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   sns3e34              ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=linux-rhel8-power9le
 -   kisdb24      ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   gnty6uq      ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   j7zn4e7          ^[email protected]%[email protected] build_system=python_pip patches=c4369ad arch=linux-rhel8-power9le
 -   dwv7fxl          ^[email protected]%[email protected] build_system=generic arch=linux-rhel8-power9le
 -   otiyzdf          ^[email protected]%[email protected] build_system=generic arch=linux-rhel8-power9le
 -   o6h5dka          ^[email protected]%[email protected] build_system=generic arch=linux-rhel8-power9le
 -   b4zqvj2      ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   dp5yjmb          ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   22kvors              ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   2kvs7lo          ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   hi2vgce              ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   vji5loq              ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   3l3rbfw                  ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   m6d5opz                  ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   scxklol                  ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   omizitc                      ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   d3ofwlg          ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   bkq42vv              ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   d3tcjlw          ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   wa6juut          ^[email protected]%[email protected]+toml build_system=python_pip arch=linux-rhel8-power9le
 -   jpxkbuu              ^[email protected]%[email protected]+man+nls+perl+subtree~svn~tcltk build_system=autotools arch=linux-rhel8-power9le
 -   jsyrgwv                  ^[email protected]%[email protected]~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rhel8-power9le
 -   3buvkan                      ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   sscceyy                  ^[email protected]%[email protected]+libbsd build_system=autotools arch=linux-rhel8-power9le
 -   lqd2k6w                      ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   semiqiq                          ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   7u7jwrh                  ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-rhel8-power9le
 -   ekntuuh                      ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=linux-rhel8-power9le
 -   2mp3nhj                      ^[email protected]%[email protected]+pic~python+shared build_system=autotools arch=linux-rhel8-power9le
 -   5vezspw                      ^[email protected]%[email protected] build_system=autotools zip=pigz arch=linux-rhel8-power9le
 -   wvhh7eg                          ^[email protected]%[email protected] build_system=makefile arch=linux-rhel8-power9le
 -   uxatbd2                          ^[email protected]%[email protected]+programs build_system=makefile compression=none libs=shared,static arch=linux-rhel8-power9le
 -   qojfcce                      ^[email protected]%[email protected]~pic build_system=autotools libs=shared,static arch=linux-rhel8-power9le
 -   ctyhzdh                  ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   l3pxd3s                      ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   nxifzhx                  ^[email protected]%[email protected]+gssapi build_system=autotools arch=linux-rhel8-power9le
 -   o5ko6ns                      ^[email protected]%[email protected]+shared build_system=autotools arch=linux-rhel8-power9le
 -   mqghkoa                          ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   erf3kcz                          ^[email protected]%[email protected] build_system=autotools patches=440b954 arch=linux-rhel8-power9le
 -   wuojhqe                      ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-power9le
 -   3klonqd                      ^[email protected]%[email protected]~obsolete_api build_system=autotools patches=4885da3 arch=linux-rhel8-power9le
 -   u6q4ryl                  ^[email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=linux-rhel8-power9le
[+]  v7vz3c5                      ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=linux-rhel8-power9le
 -   7pjgy3j                  ^[email protected]%[email protected]~jit+multibyte build_system=autotools arch=linux-rhel8-power9le
[e]  j665lil                  ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=linux-rhel8-power9le
 -   xjqodgt                  ^[email protected]%[email protected]+compat+opt build_system=autotools patches=299b958,ae9077a,b692621 arch=linux-rhel8-power9le
 -   bp36gw5              ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
 -   xae5p4w          ^[email protected]%[email protected] build_system=python_pip arch=linux-rhel8-power9le
[e]  st6e4ps      ^[email protected]%[email protected]+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,4c24573,ebdca64,f2fd060 arch=linux-rhel8-power9le
 -   lexwhm5      ^[email protected]%[email protected]+cuda~examples~exercises~ipo+openmp~rocm+shared~tests build_system=cmake build_type=Release cuda_arch=70 generator=make arch=linux-rhel8-power9le
[e]  tzrotbp      ^[email protected]%[email protected] build_system=bundle arch=linux-rhel8-power9le
 -   aqfrhkg      ^[email protected]%[email protected]+c+cuda~device_alloc~deviceconst~examples~fortran~ipo~numa~openmp~rocm~shared build_system=cmake build_type=Release cuda_arch=70 generator=make tests=none arch=linux-rhel8-power9le

==> [email protected] : has external module in ['cuda/11.4.2']
[+] /sw/ascent/cuda/11.4.2 (external cuda-11.4.2-jtm4zyvio2j5hqijt5iki6s4enionhhx)
==> [email protected] : has external module in ['cmake/3.22.2']
[+] /sw/ascent/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-10.2.0/cmake-3.22.2-obdciox7ht3vd6hzyczn732njexbhans (external cmake-3.22.2-xrvalu5csogs2on7zazr3a7mlx6pyqdw)
==> Installing cub-2.1.0-wqobtj5ffyxvtnkbfqccvhqrnute2uv3 [3/81]
==> No binary for cub-2.1.0-wqobtj5ffyxvtnkbfqccvhqrnute2uv3 found: installing from source
==> Using cached archive: /gpfs/wolf/proj-shared/csc359/exago/rcrutherford/spack-cache/source-cache/_source-cache/archive/8e/8ec47307f5e99379ac1cf6722cd5a24fc15b84b0f5361bebd453645a5e4bb34d.zip
==> Error: CommandNotFoundError: spack requires 'unzip'. Make sure it is in your path.
==> Warning: Skipping build of camp-0.2.3-fo2wqd7q2xth5xmurwercusx5axznoto since cub-2.1.0-wqobtj5ffyxvtnkbfqccvhqrnute2uv3 failed
==> Warning: Skipping build of exago-develop-kt57m336joroy6hs5oc7ieouib6g5t3v since camp-0.2.3-fo2wqd7q2xth5xmurwercusx5axznoto failed
==> Warning: Skipping build of raja-0.14.0-lexwhm55qu2b5ke5idfto24xrlhk5fm3 since camp-0.2.3-fo2wqd7q2xth5xmurwercusx5axznoto failed
==> Warning: Skipping build of hiop-develop-fsih7jodwu4jomdabvprhwtxml4cztpk since raja-0.14.0-lexwhm55qu2b5ke5idfto24xrlhk5fm3 failed
==> Warning: Skipping build of umpire-6.0.0-aqfrhkg2u5cpg5ksrbi6qjfqrxf272jf since camp-0.2.3-fo2wqd7q2xth5xmurwercusx5axznoto failed
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/gnuconfig-2022-09-17-errn6vwjuhlsnx7mnligb7u76mesi5u7
==> [email protected] : has external module in ['openblas/0.3.17-omp']
[+] /sw/ascent/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-10.2.0/openblas-0.3.17-6te4qwdzetkoyitdryljjstei6jw77gg (external openblas-0.3.17-as2v3e6fskaarmapsa45io45puojxskt)
==> [email protected] : has external module in ['spectrum-mpi/10.4.0.3-20210112']
[+] /sw/ascent/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-10.2.0/spectrum-mpi-10.4.0.3-20210112-ht5bw4jruhjujvkzcvpra5lryg5vfhy4 (external spectrum-mpi-10.4.0.3-20210112-tzrotbptuhlpdxixhwrdbit76wqzzdpg)
[+] /sw/ascent/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/perl-5.30.1-on54l56f2ipu67zkupe6nm7qyhhzn66l/ (external perl-5.30.1-j665lilcdm3e66s6il7cd72sdx33uhzp)
==> Installing zstd-1.5.5-uxatbd2chb5pvgkawcttvvkkrjau2mou [8/81]
==> No binary for zstd-1.5.5-uxatbd2chb5pvgkawcttvvkkrjau2mou found: installing from source
==> Using cached archive: /gpfs/wolf/proj-shared/csc359/exago/rcrutherford/spack-cache/source-cache/_source-cache/archive/98/98e9c3d949d1b924e28e01eccb7deed865eefebf25c2f21c702e5cd5b63b85e1.tar.gz
==> No patches needed for zstd
==> zstd: Executing phase: 'edit'
==> zstd: Executing phase: 'build'
==> zstd: Executing phase: 'install'
==> zstd: Successfully installed zstd-1.5.5-uxatbd2chb5pvgkawcttvvkkrjau2mou
  Stage: 0.25s.  Edit: 0.00s.  Build: 0.00s.  Install: 1m 30.92s.  Post-install: 0.14s.  Total: 1m 31.38s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/zstd-1.5.5-uxatbd2chb5pvgkawcttvvkkrjau2mou
[+] /sw/ascent/python/3.9/anaconda-base (external python-3.9.7-st6e4ps7i3i5tj3v7x57tvf47ddrh3or)
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/ca-certificates-mozilla-2023-05-30-v7vz3c5ckfqkz7ilw2hyzuc2oonfr6a2
==> Installing blt-0.4.1-a6k3ix4go2c45qrfvco23fkhyn7tprzl [11/81]
==> No binary for blt-0.4.1-a6k3ix4go2c45qrfvco23fkhyn7tprzl found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/16/16cc3e067ddcf48b99358107e5035a17549f52dcc701a35cd18a9d9f536826c1.tar.gz
==> No patches needed for blt
==> blt: Executing phase: 'install'
==> blt: Successfully installed blt-0.4.1-a6k3ix4go2c45qrfvco23fkhyn7tprzl
  Stage: 0.33s.  Install: 1.16s.  Post-install: 1.08s.  Total: 2.60s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/blt-0.4.1-a6k3ix4go2c45qrfvco23fkhyn7tprzl
==> Installing autoconf-archive-2023.02.20-hraabqtqfdr7bw6tsgabnyrlxbblnopa [12/81]
==> No binary for autoconf-archive-2023.02.20-hraabqtqfdr7bw6tsgabnyrlxbblnopa found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/71/71d4048479ae28f1f5794619c3d72df9c01df49b1c628ef85fde37596dc31a33.tar.xz
==> No patches needed for autoconf-archive
==> autoconf-archive: Executing phase: 'autoreconf'
==> autoconf-archive: Executing phase: 'configure'
==> autoconf-archive: Executing phase: 'build'
==> autoconf-archive: Executing phase: 'install'
==> autoconf-archive: Successfully installed autoconf-archive-2023.02.20-hraabqtqfdr7bw6tsgabnyrlxbblnopa
  Stage: 0.43s.  Autoreconf: 0.01s.  Configure: 1.44s.  Build: 0.03s.  Install: 0.56s.  Post-install: 0.94s.  Total: 3.52s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/autoconf-archive-2023.02.20-hraabqtqfdr7bw6tsgabnyrlxbblnopa
==> Installing zlib-ng-2.1.3-xjqodgty37wypez5ygiozpiqwh5iixa5 [13/81]
==> No binary for zlib-ng-2.1.3-xjqodgty37wypez5ygiozpiqwh5iixa5 found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/d2/d20e55f89d71991c59f1c5ad1ef944815e5850526c0d9cd8e504eaed5b24491a.tar.gz
==> Patch /gpfs/wolf/csc359/proj-shared/cameron/exago/tpl/spack/var/spack/repos/builtin/packages/zlib-ng/pr-1542.patch failed.
==> Error: CommandNotFoundError: spack requires 'patch'. Make sure it is in your path.
==> Warning: Skipping build of openssl-3.1.3-u6q4rylkoo6gb57ubywazls5uxkcv6zj since zlib-ng-2.1.3-xjqodgty37wypez5ygiozpiqwh5iixa5 failed
==> Warning: Skipping build of openssh-9.4p1-nxifzhxu3wgmltdyw6zjpwhetf3y33xx since openssl-3.1.3-u6q4rylkoo6gb57ubywazls5uxkcv6zj failed
==> Warning: Skipping build of git-2.41.0-jpxkbuuccdtwuhkhhl3i3lafzpnz7cld since openssh-9.4p1-nxifzhxu3wgmltdyw6zjpwhetf3y33xx failed
==> Warning: Skipping build of py-setuptools-scm-7.1.0-wa6juut5acumactlldxmixkuhiop5egi since git-2.41.0-jpxkbuuccdtwuhkhhl3i3lafzpnz7cld failed
==> Warning: Skipping build of py-hatch-vcs-0.3.0-hi2vgcerkafd6v6jfde45pghzabnpolw since py-setuptools-scm-7.1.0-wa6juut5acumactlldxmixkuhiop5egi failed
==> Warning: Skipping build of py-iniconfig-2.0.0-2kvs7lop54bpdogwax4uvu3d5jhfubb5 since py-hatch-vcs-0.3.0-hi2vgcerkafd6v6jfde45pghzabnpolw failed
==> Warning: Skipping build of py-pytest-7.3.2-b4zqvj2rmamczxuoymlv36h7pr4mfbav since py-iniconfig-2.0.0-2kvs7lop54bpdogwax4uvu3d5jhfubb5 failed
==> Warning: Skipping build of py-pluggy-1.0.0-d3tcjlwqdkrafp5j4twui5hflydcu5al since py-setuptools-scm-7.1.0-wa6juut5acumactlldxmixkuhiop5egi failed
==> Warning: Skipping build of py-hatchling-1.18.0-vji5loqcdadilzdwirmoouo7lnxe3pwk since py-pluggy-1.0.0-d3tcjlwqdkrafp5j4twui5hflydcu5al failed
==> Warning: Skipping build of py-flit-scm-1.7.0-22kvors7l4mkmg6ugk45qq6qw2ktrg7t since py-setuptools-scm-7.1.0-wa6juut5acumactlldxmixkuhiop5egi failed
==> Warning: Skipping build of py-exceptiongroup-1.1.1-dp5yjmbpc7me7w5eqko6u2lgyctxsqh7 since py-flit-scm-1.7.0-22kvors7l4mkmg6ugk45qq6qw2ktrg7t failed
==> Warning: Skipping build of krb5-1.20.1-o5ko6nsibqce3cpz4qar4doa6alguktk since openssl-3.1.3-u6q4rylkoo6gb57ubywazls5uxkcv6zj failed
==> Warning: Skipping build of curl-8.1.2-jsyrgwvimkuqdevm3me2ejgvd7vlmvet since openssl-3.1.3-u6q4rylkoo6gb57ubywazls5uxkcv6zj failed
==> Warning: Skipping build of pigz-2.7-wvhh7egylzmifsrjybtrzm3d2ydov33k since zlib-ng-2.1.3-xjqodgty37wypez5ygiozpiqwh5iixa5 failed
==> Warning: Skipping build of tar-1.34-5vezspw2x6tmaf7lnctbs2j7mh42iqdr since pigz-2.7-wvhh7egylzmifsrjybtrzm3d2ydov33k failed
==> Warning: Skipping build of gettext-0.21.1-7u7jwrhxvp7ajfsytk3ncl7tzh3rgcuv since tar-1.34-5vezspw2x6tmaf7lnctbs2j7mh42iqdr failed
==> Warning: Skipping build of bison-3.8.2-mqghkoaolgd2pb7iy54q7t3eennr4kqq since gettext-0.21.1-7u7jwrhxvp7ajfsytk3ncl7tzh3rgcuv failed
==> Warning: Skipping build of texinfo-7.0.3-5iadvvyfbq7je3u3645ocq3kpqwfxh2x since gettext-0.21.1-7u7jwrhxvp7ajfsytk3ncl7tzh3rgcuv failed
==> Warning: Skipping build of mpfr-4.2.0-ouxdolhxdx3sxhw6eqjmhkyelnmxesmi since texinfo-7.0.3-5iadvvyfbq7je3u3645ocq3kpqwfxh2x failed
==> Warning: Skipping build of suite-sparse-5.13.0-nurcezd4zysb3xeaemi74wmrmvjwz4q2 since mpfr-4.2.0-ouxdolhxdx3sxhw6eqjmhkyelnmxesmi failed
==> Warning: Skipping build of libxml2-2.10.3-2mp3nhjvdod5gh5yra4csxldfznxb4mr since zlib-ng-2.1.3-xjqodgty37wypez5ygiozpiqwh5iixa5 failed
==> Installing findutils-4.9.0-erf3kczlfv6glig2agkp2dskqa5asweu [14/81]
==> No binary for findutils-4.9.0-erf3kczlfv6glig2agkp2dskqa5asweu found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/a2/a2bfb8c09d436770edc59f50fa483e785b161a3b7b9d547573cb08065fd462fe.tar.xz
==> Patch /gpfs/wolf/csc359/proj-shared/cameron/exago/tpl/spack/var/spack/repos/builtin/packages/findutils/nonnull.patch failed.
==> Error: CommandNotFoundError: spack requires 'patch'. Make sure it is in your path.
==> Installing pkgconf-1.9.5-kisdb24dfkb2m4z5d7mcx5ykb3mdyial [15/81]
==> No binary for pkgconf-1.9.5-kisdb24dfkb2m4z5d7mcx5ykb3mdyial found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/1a/1ac1656debb27497563036f7bffc281490f83f9b8457c0d60bcfb638fb6b6171.tar.xz
==> No patches needed for pkgconf
==> pkgconf: Executing phase: 'autoreconf'
==> pkgconf: Executing phase: 'configure'
==> pkgconf: Executing phase: 'build'
==> pkgconf: Executing phase: 'install'
==> pkgconf: Successfully installed pkgconf-1.9.5-kisdb24dfkb2m4z5d7mcx5ykb3mdyial
  Stage: 0.15s.  Autoreconf: 0.06s.  Configure: 3.63s.  Build: 4.91s.  Install: 0.28s.  Post-install: 0.16s.  Total: 9.29s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/pkgconf-1.9.5-kisdb24dfkb2m4z5d7mcx5ykb3mdyial
==> Installing libiconv-1.17-sns3e34mm76ikhlsqdfme2ifjtjz7z63 [16/81]
==> No binary for libiconv-1.17-sns3e34mm76ikhlsqdfme2ifjtjz7z63 found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/8f/8f74213b56238c85a50a5329f77e06198771e70dd9a739779f4c02f65d971313.tar.gz
==> No patches needed for libiconv
==> libiconv: Executing phase: 'autoreconf'
==> libiconv: Executing phase: 'configure'
==> libiconv: Executing phase: 'build'
==> libiconv: Executing phase: 'install'
==> libiconv: Successfully installed libiconv-1.17-sns3e34mm76ikhlsqdfme2ifjtjz7z63
  Stage: 0.65s.  Autoreconf: 0.13s.  Configure: 16.88s.  Build: 16.62s.  Install: 0.85s.  Post-install: 0.32s.  Total: 35.54s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/libiconv-1.17-sns3e34mm76ikhlsqdfme2ifjtjz7z63
==> Installing libsigsegv-2.14-yomz2kehbanl235tzljtlcpm767svvmj [17/81]
==> No binary for libsigsegv-2.14-yomz2kehbanl235tzljtlcpm767svvmj found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/cd/cdac3941803364cf81a908499beb79c200ead60b6b5b40cad124fd1e06caa295.tar.gz
==> No patches needed for libsigsegv
==> libsigsegv: Executing phase: 'autoreconf'
==> libsigsegv: Executing phase: 'configure'
==> libsigsegv: Executing phase: 'build'
==> libsigsegv: Executing phase: 'install'
==> libsigsegv: Successfully installed libsigsegv-2.14-yomz2kehbanl235tzljtlcpm767svvmj
  Stage: 0.16s.  Autoreconf: 0.06s.  Configure: 6.02s.  Build: 1.69s.  Install: 0.20s.  Post-install: 0.13s.  Total: 8.36s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/libsigsegv-2.14-yomz2kehbanl235tzljtlcpm767svvmj
==> Installing xz-5.4.1-qojfccejcd7ubks3wdi43gldjlbttcdw [18/81]
==> No binary for xz-5.4.1-qojfccejcd7ubks3wdi43gldjlbttcdw found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/dd/dd172acb53867a68012f94c17389401b2f274a1aa5ae8f84cbfb8b7e383ea8d3.tar.bz2
==> No patches needed for xz
==> xz: Executing phase: 'autoreconf'
==> xz: Executing phase: 'configure'
==> xz: Executing phase: 'build'
==> xz: Executing phase: 'install'
==> xz: Successfully installed xz-5.4.1-qojfccejcd7ubks3wdi43gldjlbttcdw
  Stage: 0.51s.  Autoreconf: 0.08s.  Configure: 9.37s.  Build: 32.69s.  Install: 2.06s.  Post-install: 0.42s.  Total: 45.22s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/xz-5.4.1-qojfccejcd7ubks3wdi43gldjlbttcdw
==> Installing pcre2-10.42-7pjgy3je6s2peducvdxn3jko4htvypmw [19/81]
==> No binary for pcre2-10.42-7pjgy3je6s2peducvdxn3jko4htvypmw found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/8d/8d36cd8cb6ea2a4c2bb358ff6411b0c788633a2a45dabbf1aeb4b701d1b5e840.tar.bz2
==> No patches needed for pcre2
==> pcre2: Executing phase: 'autoreconf'
==> pcre2: Executing phase: 'configure'
==> pcre2: Executing phase: 'build'
==> pcre2: Executing phase: 'install'
==> pcre2: Successfully installed pcre2-10.42-7pjgy3je6s2peducvdxn3jko4htvypmw
  Stage: 0.50s.  Autoreconf: 0.06s.  Configure: 4.92s.  Build: 1m 16.40s.  Install: 0.69s.  Post-install: 0.40s.  Total: 1m 23.07s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/pcre2-10.42-7pjgy3je6s2peducvdxn3jko4htvypmw
==> Installing gmake-4.4.1-hrielr2sbwinzaxnludykps2ubij3mdm [20/81]
==> No binary for gmake-4.4.1-hrielr2sbwinzaxnludykps2ubij3mdm found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/dd/dd16fb1d67bfab79a72f5e8390735c49e3e8e70b4945a15ab1f81ddb78658fb3.tar.gz
==> No patches needed for gmake
==> gmake: Executing phase: 'autoreconf'
==> gmake: Executing phase: 'configure'
==> gmake: Executing phase: 'build'
==> gmake: Executing phase: 'install'
==> gmake: Successfully installed gmake-4.4.1-hrielr2sbwinzaxnludykps2ubij3mdm
  Stage: 0.27s.  Autoreconf: 0.07s.  Configure: 9.46s.  Build: 10.63s.  Install: 0.01s.  Post-install: 0.11s.  Total: 20.63s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/gmake-4.4.1-hrielr2sbwinzaxnludykps2ubij3mdm
==> Installing libmd-1.0.4-semiqiq4gly5l6zlfl5bkeni47gjqwcr [21/81]
==> No binary for libmd-1.0.4-semiqiq4gly5l6zlfl5bkeni47gjqwcr found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/f5/f51c921042e34beddeded4b75557656559cf5b1f2448033b4c1eec11c07e530f.tar.xz
==> No patches needed for libmd
==> libmd: Executing phase: 'autoreconf'
==> libmd: Executing phase: 'configure'
==> libmd: Executing phase: 'build'
==> libmd: Executing phase: 'install'
==> libmd: Successfully installed libmd-1.0.4-semiqiq4gly5l6zlfl5bkeni47gjqwcr
  Stage: 0.17s.  Autoreconf: 0.07s.  Configure: 4.87s.  Build: 6.44s.  Install: 0.30s.  Post-install: 0.18s.  Total: 12.13s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/libmd-1.0.4-semiqiq4gly5l6zlfl5bkeni47gjqwcr
==> Installing coinhsl-2019.05.21-tujzksucszd4wl4b2fvzpvxaucwy4jqk [22/81]
==> No binary for coinhsl-2019.05.21-tujzksucszd4wl4b2fvzpvxaucwy4jqk found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/95/95ce1160f0b013151a3e25d40337775c760a8f3a79d801a1d190598bf4e4c0c3.tar.gz
==> No patches needed for coinhsl
==> coinhsl: Executing phase: 'autoreconf'
==> coinhsl: Executing phase: 'configure'
==> coinhsl: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j1' 'V=1'

2 errors found in build log:
     144    ==> [2023-09-29-17:40:28.705999] FILTER FILE: /gpfs/wolf/proj-share
            d/csc359/exago/rcrutherford/spack-cache/spack-stage/spack-stage-coi
            nhsl-2019.05.21-tujzksucszd4wl4b2fvzpvxaucwy4jqk/spack-src/libtool
            [replacing "^pic_flag=""$"]
     145    ==> [2023-09-29-17:40:28.809206] FILTER FILE: /gpfs/wolf/proj-share
            d/csc359/exago/rcrutherford/spack-cache/spack-stage/spack-stage-coi
            nhsl-2019.05.21-tujzksucszd4wl4b2fvzpvxaucwy4jqk/spack-src/libtool
            [replacing "^pic_flag=""$"]
     146    ==> coinhsl: Executing phase: 'build'
     147    ==> [2023-09-29-17:40:28.927532] 'make' '-j1' 'V=1'
     148    (CDPATH="${ZSH_VERSION+.}:" && cd /gpfs/wolf/proj-shared/csc359/exa
            go/rcrutherford/spack-cache/spack-stage/spack-stage-coinhsl-2019.05
            .21-tujzksucszd4wl4b2fvzpvxaucwy4jqk/spack-src && /bin/sh /gpfs/wol
            f/proj-shared/csc359/exago/rcrutherford/spack-cache/spack-stage/spa
            ck-stage-coinhsl-2019.05.21-tujzksucszd4wl4b2fvzpvxaucwy4jqk/spack-
            src/missing autoheader)
     149    /gpfs/wolf/proj-shared/csc359/exago/rcrutherford/spack-cache/spack-
            stage/spack-stage-coinhsl-2019.05.21-tujzksucszd4wl4b2fvzpvxaucwy4j
            qk/spack-src/missing: line 81: autoheader: command not found
  >> 150    WARNING: 'autoheader' is missing on your system.
     151             You should only need it if you modified 'acconfig.h' or
     152             'configure.ac' or m4 files included by 'configure.ac'.
     153             The 'autoheader' program is part of the GNU Autoconf packa
            ge:
     154             <http://www.gnu.org/software/autoconf/>
     155             It also requires GNU m4 and Perl in order to run:
     156             <http://www.gnu.org/software/m4/>
     157             <http://www.perl.org/>
  >> 158    make: *** [Makefile:596: /gpfs/wolf/proj-shared/csc359/exago/rcruth
            erford/spack-cache/spack-stage/spack-stage-coinhsl-2019.05.21-tujzk
            sucszd4wl4b2fvzpvxaucwy4jqk/spack-src/config.h.in] Error 127

See build log for details:
  /gpfs/wolf/proj-shared/csc359/exago/rcrutherford/spack-cache/spack-stage/spack-stage-coinhsl-2019.05.21-tujzksucszd4wl4b2fvzpvxaucwy4jqk/spack-build-out.txt

==> Warning: Skipping build of ipopt-3.12.10-sgutdp7mdf5epnusbxefcpyycc25kc74 since coinhsl-2019.05.21-tujzksucszd4wl4b2fvzpvxaucwy4jqk failed
==> Installing libxcrypt-4.4.35-3klonqdrvfppx7qiylgibg5yawzlrm7k [23/81]
==> No binary for libxcrypt-4.4.35-3klonqdrvfppx7qiylgibg5yawzlrm7k found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/a8/a8c935505b55f1df0d17f8bfd59468c7c6709a1d31831b0f8e3e045ab8fd455d.tar.xz
==> Patch /gpfs/wolf/csc359/proj-shared/cameron/exago/tpl/spack/var/spack/repos/builtin/packages/libxcrypt/commit-95d56e0.patch failed.
==> Error: CommandNotFoundError: spack requires 'patch'. Make sure it is in your path.
==> Installing py-pip-23.1.2-dwv7fxlei5unu2ftofmadht73xwgzdvq [24/81]
==> No binary for py-pip-23.1.2-dwv7fxlei5unu2ftofmadht73xwgzdvq found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/3e/3ef6ac33239e4027d9a5598a381b9d30880a1477e50039db2eac6e8a8f6d1b18
==> No patches needed for py-pip
==> py-pip: Executing phase: 'install'
==> py-pip: Successfully installed py-pip-23.1.2-dwv7fxlei5unu2ftofmadht73xwgzdvq
  Stage: 0.12s.  Install: 6.58s.  Post-install: 1.51s.  Total: 8.24s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/py-pip-23.1.2-dwv7fxlei5unu2ftofmadht73xwgzdvq
==> Installing nghttp2-1.52.0-3buvkann72edx7c3yzypjeiahcstllqf [25/81]
==> No binary for nghttp2-1.52.0-3buvkann72edx7c3yzypjeiahcstllqf found: installing from source
==> Fetching file:///gpfs/wolf/csc359/world-shared/exago/spack-ci/mirror/_source-cache/archive/98/9877caa62bd72dde1331da38ce039dadb049817a01c3bdee809da15b754771b8.tar.gz
==> No patches needed for nghttp2
==> nghttp2: Executing phase: 'autoreconf'
==> nghttp2: Executing phase: 'configure'
==> nghttp2: Executing phase: 'build'
==> nghttp2: Executing phase: 'install'
==> nghttp2: Successfully installed nghttp2-1.52.0-3buvkann72edx7c3yzypjeiahcstllqf
  Stage: 1.14s.  Autoreconf: 0.15s.  Configure: 12.61s.  Build: 10.08s.  Install: 0.39s.  Post-install: 0.15s.  Total: 24.61s
[+] /gpfs/wolf/proj-shared/csc359/exago/spack-ci/install/linux-rhel8-power9le/gcc-10.2.0/nghttp2-1.52.0-3buvkann72edx7c3yzypjeiahcstllqf

Seems like as well there are some weird system issues happening here... Will have to investigate further

@nkoukpaizan
Copy link
Collaborator Author

It looks like unzip is not found on the compute nodes. jsrun -n1 which unzip returns which: no unzip in ...

@nkoukpaizan
Copy link
Collaborator Author

Spack also complains about autoheader not being found during the coinhsl build. A diff between jsrun -n1 ls /usr/bin and ls /usr/bin in an interactive session shows several differences; not just unzip and autoheader.

@cameronrutherford
Copy link
Contributor

Spack also complains about autoheader not being found during the coinhsl build. A diff between jsrun -n1 ls /usr/bin and ls /usr/bin in an interactive session shows several differences; not just unzip and autoheader.

Hmm well I am not quite sure I understand the path forward. Should I be then just installing everything in the nobatch parts of ORNL CI where I would have access to these libraries, or is there a way to converge the environments between the batch and compute nodes?

@cameronrutherford
Copy link
Contributor

cameronrutherford commented Oct 2, 2023

I think building on the batch node should be fine, even though we really should be using compute node instances. Would there be an issue with just doing spack builds in the nobatch queue from ORNL's perspective in CI?

@rothpc
Copy link
Collaborator

rothpc commented Oct 2, 2023

It appears the ascent compute nodes don't have all the same RPMs installed as on the login node(s). I see at least two paths forward:

  • specify the software in your Spack environment and build it as part of the environment if not told by the config that it is external.
  • update your MODULEPATH to add the appropriate path like /sw/ascent/spack-envs/base/modules/spack/linux-rhel8-ppc64le/Core, then do 'module load's for the software that you find is missing.

@cameronrutherford
Copy link
Contributor

@rothpc and @nkoukpaizan wolf is down today, but I was able to get a working ExaGO build last night working around quite a few different bugs.

Unzip was configurable through a module. Autoheader was another module. Difficult one was patch was not available on the compute node, so I had to use spack to install that, load it in, and then install remaining packages.

Other bug is that when hiop is cloned into the source mirror, for some reason submodules are not cloned. This resulted in build errors for eigen. IMO we should consider moving eigen to a spack managed dependency, but that can happen after we get things working for exago. Fix to this issue was cloning hiop in the build with spack develop --clone -f FORCE --path=$(pwd)/hiop hiop@develop, and then manually initializing submodules.`

It's possible that using the correct HTTP proxy configuration means we don't need a source mirror configured, so I want to try that out briefly.

Apart from that, only remaining step is to put pipelines in place. Thanks for all the help!

@cameronrutherford
Copy link
Contributor

For some reason there was a glibc issue compiling exago with PETSc using gcc 10.2.0 and cuda 11.4 on the system. I tried to instead build with gcc 11.2.0 and a spack built cuda 11.8 to see if that gives us any more luck (since just 11.2 wouldn't work as cuda 11.4 is incompatible with this compiler).

buildsystem/gcc-cuda/ascent/base.sh Outdated Show resolved Hide resolved
buildsystem/gcc-cuda/ascent/base.sh Outdated Show resolved Hide resolved
@cameronrutherford
Copy link
Contributor

@nkoukpaizan have you seen the error in this pipeline before? https://code.ornl.gov/ecpcitest/exasgd/exago/-/jobs/2070906

As far as I can tell I rebuilt all the PETSc dependencies with the latest compiler, so not sure where this issue is coming from.

Maybe I broke something you had already fixed? If you push to the branch, try out the auto rebuild by adding [ascent-rebuild] in your commit message.

I am likely thinking my mix of external packages is causing issues, but perhaps this is actually a bug with the newest PETSc version?

@cameronrutherford cameronrutherford force-pushed the nicholson/upgrade-ascent-buildsystem branch from 05319c3 to de283f0 Compare October 11, 2023 19:52
@cameronrutherford cameronrutherford force-pushed the nicholson/upgrade-ascent-buildsystem branch from 741ce9a to 1516a74 Compare October 12, 2023 16:07
@cameronrutherford
Copy link
Contributor

I removed the change for compatibility with hiop@develop, and have pinned 1.0.0 since that is the latest release that we can build with. Builds are churning on all platforms

@cameronrutherford cameronrutherford linked an issue Oct 12, 2023 that may be closed by this pull request
@cameronrutherford
Copy link
Contributor

CI passing on Newell and Incline is a good sign. Will have to wait for Ascent to re-build Python to verify that is fixed, but things are looking good.

This certainly clarifies that [email protected] will require [email protected], and we will look to fix @develop capability after the release

@cameronrutherford
Copy link
Contributor

Ascent test is running - if that passes (along with passing status indications from other platforms, I think we are ready to merge!

CHANGELOG.md Show resolved Hide resolved
tests/unit/CMakeLists.txt Show resolved Hide resolved
@cameronrutherford cameronrutherford merged commit ea2f3cb into develop Oct 12, 2023
7 checks passed
bjpalmer pushed a commit that referenced this pull request Oct 16, 2023
* Boilerplate scripts to install modules on Ascent via submodule Spack

* Fix '--nnodes'-->'-nodes' on Ascent

* Improve Ascent env.sh

* [email protected] on Ascent

* Apply pre-commmit fixes

* Relax constraints on exago dependencies on Ascent and build ~python

* concretizer: reuse was causing several packages to be duplicated in the environment. Require clean concretizations on  Ascent.

* Minor module update on Ascent

* Add LAPACK_LIBRARIES to Ascent base script. CMAKE was picking up python's openblas otherwise.

* Error with unzip.

* Apply pre-commmit fixes

* Add working build on ascent.

* Add working gcc11.2.0 spack spec.

* Add Ascent Spack pipeline. [ascent-rebuild]

* Update gcc version to 11.2.0 in base.sh [skip-ci]

* Fix stages of Ascent pipeline [ascent-rebuild]

* Add working ascent spack build.

* Add hiop@develop force rebuild to PNNL CI [ascent-rebuild] [newell-rebuild] [deception-rebuild] [incline-rebuild].

* Update Ascent spack built tcl modules

* Only test ascent on tcl module update [ci-skip]

* Update base.sh to disable python on ascent [skip ci]

* Remove LAPACK_LIBRARIES spec [ascent-test]

* Update ascent.gitlab-ci.yml to fix needs/dependencies [ascent-test]

* Update deception spack built tcl modules - [deception-test]

* Try again with Python, but have Spack build it instead of using the external module [ascent-rebuild]

* Force python rebuild on ascent and use [email protected] on incline [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Pin [email protected] on all CI platforms [decetpion-rebuild] [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Fix false positive/negative in Ascent pipelines [deception-rebuild] [ascent-test]

* Update incline spack built tcl modules - [incline-test]

* Update newell spack built tcl modules - [newell-test]

* Fix HiOp spec on Ascent [ascent-rebuild].

* Update deception spack built tcl modules - [deception-test]

* Update CPU Spack build with issue for each failing build [ci skip]

* Update Ascent spack built tcl modules [ascent-test]

* Add 1.0.0 dep into CHANGELOG.

* Add ascent-skip to CI to get tests passing [ascent-test]

---------

Co-authored-by: nkoukpaizan <[email protected]>
Co-authored-by: Cameron Rutherford <[email protected]>
Co-authored-by: cameronrutherford <[email protected]>
Co-authored-by: spack-auto-module <[email protected]>
cameronrutherford added a commit that referenced this pull request Oct 25, 2023
* Boilerplate scripts to install modules on Ascent via submodule Spack

* Fix '--nnodes'-->'-nodes' on Ascent

* Improve Ascent env.sh

* [email protected] on Ascent

* Apply pre-commmit fixes

* Relax constraints on exago dependencies on Ascent and build ~python

* concretizer: reuse was causing several packages to be duplicated in the environment. Require clean concretizations on  Ascent.

* Minor module update on Ascent

* Add LAPACK_LIBRARIES to Ascent base script. CMAKE was picking up python's openblas otherwise.

* Error with unzip.

* Apply pre-commmit fixes

* Add working build on ascent.

* Add working gcc11.2.0 spack spec.

* Add Ascent Spack pipeline. [ascent-rebuild]

* Update gcc version to 11.2.0 in base.sh [skip-ci]

* Fix stages of Ascent pipeline [ascent-rebuild]

* Add working ascent spack build.

* Add hiop@develop force rebuild to PNNL CI [ascent-rebuild] [newell-rebuild] [deception-rebuild] [incline-rebuild].

* Update Ascent spack built tcl modules

* Only test ascent on tcl module update [ci-skip]

* Update base.sh to disable python on ascent [skip ci]

* Remove LAPACK_LIBRARIES spec [ascent-test]

* Update ascent.gitlab-ci.yml to fix needs/dependencies [ascent-test]

* Update deception spack built tcl modules - [deception-test]

* Try again with Python, but have Spack build it instead of using the external module [ascent-rebuild]

* Force python rebuild on ascent and use [email protected] on incline [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Pin [email protected] on all CI platforms [decetpion-rebuild] [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Fix false positive/negative in Ascent pipelines [deception-rebuild] [ascent-test]

* Update incline spack built tcl modules - [incline-test]

* Update newell spack built tcl modules - [newell-test]

* Fix HiOp spec on Ascent [ascent-rebuild].

* Update deception spack built tcl modules - [deception-test]

* Update CPU Spack build with issue for each failing build [ci skip]

* Update Ascent spack built tcl modules [ascent-test]

* Add 1.0.0 dep into CHANGELOG.

* Add ascent-skip to CI to get tests passing [ascent-test]

---------

Co-authored-by: nkoukpaizan <[email protected]>
Co-authored-by: Cameron Rutherford <[email protected]>
Co-authored-by: cameronrutherford <[email protected]>
Co-authored-by: spack-auto-module <[email protected]>
bjpalmer pushed a commit that referenced this pull request Oct 30, 2023
* Boilerplate scripts to install modules on Ascent via submodule Spack

* Fix '--nnodes'-->'-nodes' on Ascent

* Improve Ascent env.sh

* [email protected] on Ascent

* Apply pre-commmit fixes

* Relax constraints on exago dependencies on Ascent and build ~python

* concretizer: reuse was causing several packages to be duplicated in the environment. Require clean concretizations on  Ascent.

* Minor module update on Ascent

* Add LAPACK_LIBRARIES to Ascent base script. CMAKE was picking up python's openblas otherwise.

* Error with unzip.

* Apply pre-commmit fixes

* Add working build on ascent.

* Add working gcc11.2.0 spack spec.

* Add Ascent Spack pipeline. [ascent-rebuild]

* Update gcc version to 11.2.0 in base.sh [skip-ci]

* Fix stages of Ascent pipeline [ascent-rebuild]

* Add working ascent spack build.

* Add hiop@develop force rebuild to PNNL CI [ascent-rebuild] [newell-rebuild] [deception-rebuild] [incline-rebuild].

* Update Ascent spack built tcl modules

* Only test ascent on tcl module update [ci-skip]

* Update base.sh to disable python on ascent [skip ci]

* Remove LAPACK_LIBRARIES spec [ascent-test]

* Update ascent.gitlab-ci.yml to fix needs/dependencies [ascent-test]

* Update deception spack built tcl modules - [deception-test]

* Try again with Python, but have Spack build it instead of using the external module [ascent-rebuild]

* Force python rebuild on ascent and use [email protected] on incline [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Pin [email protected] on all CI platforms [decetpion-rebuild] [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Fix false positive/negative in Ascent pipelines [deception-rebuild] [ascent-test]

* Update incline spack built tcl modules - [incline-test]

* Update newell spack built tcl modules - [newell-test]

* Fix HiOp spec on Ascent [ascent-rebuild].

* Update deception spack built tcl modules - [deception-test]

* Update CPU Spack build with issue for each failing build [ci skip]

* Update Ascent spack built tcl modules [ascent-test]

* Add 1.0.0 dep into CHANGELOG.

* Add ascent-skip to CI to get tests passing [ascent-test]

---------

Co-authored-by: nkoukpaizan <[email protected]>
Co-authored-by: Cameron Rutherford <[email protected]>
Co-authored-by: cameronrutherford <[email protected]>
Co-authored-by: spack-auto-module <[email protected]>
bjpalmer pushed a commit that referenced this pull request Nov 9, 2023
* Boilerplate scripts to install modules on Ascent via submodule Spack

* Fix '--nnodes'-->'-nodes' on Ascent

* Improve Ascent env.sh

* [email protected] on Ascent

* Apply pre-commmit fixes

* Relax constraints on exago dependencies on Ascent and build ~python

* concretizer: reuse was causing several packages to be duplicated in the environment. Require clean concretizations on  Ascent.

* Minor module update on Ascent

* Add LAPACK_LIBRARIES to Ascent base script. CMAKE was picking up python's openblas otherwise.

* Error with unzip.

* Apply pre-commmit fixes

* Add working build on ascent.

* Add working gcc11.2.0 spack spec.

* Add Ascent Spack pipeline. [ascent-rebuild]

* Update gcc version to 11.2.0 in base.sh [skip-ci]

* Fix stages of Ascent pipeline [ascent-rebuild]

* Add working ascent spack build.

* Add hiop@develop force rebuild to PNNL CI [ascent-rebuild] [newell-rebuild] [deception-rebuild] [incline-rebuild].

* Update Ascent spack built tcl modules

* Only test ascent on tcl module update [ci-skip]

* Update base.sh to disable python on ascent [skip ci]

* Remove LAPACK_LIBRARIES spec [ascent-test]

* Update ascent.gitlab-ci.yml to fix needs/dependencies [ascent-test]

* Update deception spack built tcl modules - [deception-test]

* Try again with Python, but have Spack build it instead of using the external module [ascent-rebuild]

* Force python rebuild on ascent and use [email protected] on incline [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Pin [email protected] on all CI platforms [decetpion-rebuild] [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Fix false positive/negative in Ascent pipelines [deception-rebuild] [ascent-test]

* Update incline spack built tcl modules - [incline-test]

* Update newell spack built tcl modules - [newell-test]

* Fix HiOp spec on Ascent [ascent-rebuild].

* Update deception spack built tcl modules - [deception-test]

* Update CPU Spack build with issue for each failing build [ci skip]

* Update Ascent spack built tcl modules [ascent-test]

* Add 1.0.0 dep into CHANGELOG.

* Add ascent-skip to CI to get tests passing [ascent-test]

---------

Co-authored-by: nkoukpaizan <[email protected]>
Co-authored-by: Cameron Rutherford <[email protected]>
Co-authored-by: cameronrutherford <[email protected]>
Co-authored-by: spack-auto-module <[email protected]>
bjpalmer pushed a commit that referenced this pull request Nov 20, 2023
* Boilerplate scripts to install modules on Ascent via submodule Spack

* Fix '--nnodes'-->'-nodes' on Ascent

* Improve Ascent env.sh

* [email protected] on Ascent

* Apply pre-commmit fixes

* Relax constraints on exago dependencies on Ascent and build ~python

* concretizer: reuse was causing several packages to be duplicated in the environment. Require clean concretizations on  Ascent.

* Minor module update on Ascent

* Add LAPACK_LIBRARIES to Ascent base script. CMAKE was picking up python's openblas otherwise.

* Error with unzip.

* Apply pre-commmit fixes

* Add working build on ascent.

* Add working gcc11.2.0 spack spec.

* Add Ascent Spack pipeline. [ascent-rebuild]

* Update gcc version to 11.2.0 in base.sh [skip-ci]

* Fix stages of Ascent pipeline [ascent-rebuild]

* Add working ascent spack build.

* Add hiop@develop force rebuild to PNNL CI [ascent-rebuild] [newell-rebuild] [deception-rebuild] [incline-rebuild].

* Update Ascent spack built tcl modules

* Only test ascent on tcl module update [ci-skip]

* Update base.sh to disable python on ascent [skip ci]

* Remove LAPACK_LIBRARIES spec [ascent-test]

* Update ascent.gitlab-ci.yml to fix needs/dependencies [ascent-test]

* Update deception spack built tcl modules - [deception-test]

* Try again with Python, but have Spack build it instead of using the external module [ascent-rebuild]

* Force python rebuild on ascent and use [email protected] on incline [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Pin [email protected] on all CI platforms [decetpion-rebuild] [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Fix false positive/negative in Ascent pipelines [deception-rebuild] [ascent-test]

* Update incline spack built tcl modules - [incline-test]

* Update newell spack built tcl modules - [newell-test]

* Fix HiOp spec on Ascent [ascent-rebuild].

* Update deception spack built tcl modules - [deception-test]

* Update CPU Spack build with issue for each failing build [ci skip]

* Update Ascent spack built tcl modules [ascent-test]

* Add 1.0.0 dep into CHANGELOG.

* Add ascent-skip to CI to get tests passing [ascent-test]

---------

Co-authored-by: nkoukpaizan <[email protected]>
Co-authored-by: Cameron Rutherford <[email protected]>
Co-authored-by: cameronrutherford <[email protected]>
Co-authored-by: spack-auto-module <[email protected]>
cameronrutherford added a commit that referenced this pull request Nov 28, 2023
* Boilerplate scripts to install modules on Ascent via submodule Spack

* Fix '--nnodes'-->'-nodes' on Ascent

* Improve Ascent env.sh

* [email protected] on Ascent

* Apply pre-commmit fixes

* Relax constraints on exago dependencies on Ascent and build ~python

* concretizer: reuse was causing several packages to be duplicated in the environment. Require clean concretizations on  Ascent.

* Minor module update on Ascent

* Add LAPACK_LIBRARIES to Ascent base script. CMAKE was picking up python's openblas otherwise.

* Error with unzip.

* Apply pre-commmit fixes

* Add working build on ascent.

* Add working gcc11.2.0 spack spec.

* Add Ascent Spack pipeline. [ascent-rebuild]

* Update gcc version to 11.2.0 in base.sh [skip-ci]

* Fix stages of Ascent pipeline [ascent-rebuild]

* Add working ascent spack build.

* Add hiop@develop force rebuild to PNNL CI [ascent-rebuild] [newell-rebuild] [deception-rebuild] [incline-rebuild].

* Update Ascent spack built tcl modules

* Only test ascent on tcl module update [ci-skip]

* Update base.sh to disable python on ascent [skip ci]

* Remove LAPACK_LIBRARIES spec [ascent-test]

* Update ascent.gitlab-ci.yml to fix needs/dependencies [ascent-test]

* Update deception spack built tcl modules - [deception-test]

* Try again with Python, but have Spack build it instead of using the external module [ascent-rebuild]

* Force python rebuild on ascent and use [email protected] on incline [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Pin [email protected] on all CI platforms [decetpion-rebuild] [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Fix false positive/negative in Ascent pipelines [deception-rebuild] [ascent-test]

* Update incline spack built tcl modules - [incline-test]

* Update newell spack built tcl modules - [newell-test]

* Fix HiOp spec on Ascent [ascent-rebuild].

* Update deception spack built tcl modules - [deception-test]

* Update CPU Spack build with issue for each failing build [ci skip]

* Update Ascent spack built tcl modules [ascent-test]

* Add 1.0.0 dep into CHANGELOG.

* Add ascent-skip to CI to get tests passing [ascent-test]

---------

Co-authored-by: nkoukpaizan <[email protected]>
Co-authored-by: Cameron Rutherford <[email protected]>
Co-authored-by: cameronrutherford <[email protected]>
Co-authored-by: spack-auto-module <[email protected]>
cameronrutherford added a commit that referenced this pull request Nov 29, 2023
* only print error messages if mpi rank is 0

* add rank check for num ranks

* have non-zero ranks exit gracefully when throwing exago error

* pflow functionality tests fully mpi aware

* add logging rank variable

* Apply pre-commmit fixes

* Deleted unused header file.

* Brought SCOPFLOW test driver in line with PFLOW driver.

* Applied additional changes to selfcheck.cpp file for PFLOW, SOPFLOW and SCOPFLOW
to adapt tests for running on multiple MPI ranks.

* Apply pre-commmit fixes

* Initialized some variables that were not getting properly set for serial
test case.

* only print error messages if mpi rank is 0

* pflow functionality tests fully mpi aware

* add logging rank variable

* Apply pre-commmit fixes

* Deleted unused header file.

* Brought SCOPFLOW test driver in line with PFLOW driver.

* Applied additional changes to selfcheck.cpp file for PFLOW, SOPFLOW and SCOPFLOW
to adapt tests for running on multiple MPI ranks.

* Apply pre-commmit fixes

* Update summit modules (#21)

* Minor fix for Summit build system

* Fix '--nnodes'-->'-nodes' on Summit

* Attempt to update Summit modules

* Reinstall Ginkgo and python dependencies on Summit

* Enforce [email protected] on Summit

* Specify RelWithDebInfo for ExaGO and HiOp on Summit

* Update Spack

* Relax constraints on exago dependencies on Summit

* Add constraints on HiOp in the spack config. Part of the ExaGO package was conflicting with building HiOp in release mode.

* Cleaner module install on Summit

* Update spack_cpu_build.yaml to work without fork

* Update .github/workflows/spack_cpu_build.yaml

* Update Spack

* Try updating pybind11 submodule to see if it fixes errors with exago+python builds

---------

Co-authored-by: Cameron Rutherford <[email protected]>

* OPFLOW with RAJA/HIOP sparse GPU solvers (#8)

* OPFLOW: initial implementation of RAJA/HiOp sparse GPU-based solver

WIP - HIOP Sparse solver with GPU model

OPFLOW: Started work on support for HIOP sparse solver interface for GPUs.

Added a copy of hiop sparse solver interface.

OPFLOW: Added model skeleton for GPU sparse version (copying from pbpolrajahiop)

Fixed build

Did some copy paste to add a test for HIOPSPARSE. This test is not actually
functional yet.

Started updating the hiopsparse model and solver code.

More work on updating the solver and model

Added scalar and vector unit tests for model to be used with HIOP sparse solver on GPU

Apply cmake lint

Fix unit tests.

Set the size of array when using Umpire memset.

Code formatting

Some minor changes to get PBPOLRAJAHIOPSPARSE model code to compile

Separate BUS/LINE/GEN/.../Param structs into reusable module

Minor edit

Rename files

Fix typo

Use BUS/LINE/GEN/.../Param structs in Raja HiOp Sparse model (compiles)

Updating HIOP sparse solver GPU API

Completed bounds kernels

Completed scalar and vector functions

WIP - HIOP Sparse solver with GPU model

OPFLOW: Started work on support for HIOP sparse solver interface for GPUs.

Added a copy of hiop sparse solver interface.

OPFLOW: Added model skeleton for GPU sparse version (copying from pbpolrajahiop)

Fixed build

Did some copy paste to add a test for HIOPSPARSE. This test is not actually
functional yet.

Started updating the hiopsparse model and solver code.

More work on updating the solver and model

Added scalar and vector unit tests for model to be used with HIOP sparse solver on GPU

Apply cmake lint

Fix unit tests.

Set the size of array when using Umpire memset.

Code formatting

Rename files

Use BUS/LINE/GEN/.../Param structs in Raja HiOp Sparse model (compiles)

Updating HIOP sparse solver GPU API

Completed bounds kernels

Jacobian and Hessian for sparse model (CPU --> GPU copy)

Use correct array lengths in Eq. Jacobian

Fix bug in Jacobian.

Fix unused variable/parameter errors

OPFLOW: rework solution callback for RAJA/HIOP GPU-based solver

Formatting changes

* Add unit test for RAJA/HiOp Sparse GPU model (9-bus only)

* Apply pre-commmit fixes

* Add test for 200-bus case

* Apply pre-commmit fixes

---------

Co-authored-by: Abhyankar, Shrirang G <[email protected]>

* Upgrade ascent build system and use `[email protected]` on CI platforms (#20)

* Boilerplate scripts to install modules on Ascent via submodule Spack

* Fix '--nnodes'-->'-nodes' on Ascent

* Improve Ascent env.sh

* [email protected] on Ascent

* Apply pre-commmit fixes

* Relax constraints on exago dependencies on Ascent and build ~python

* concretizer: reuse was causing several packages to be duplicated in the environment. Require clean concretizations on  Ascent.

* Minor module update on Ascent

* Add LAPACK_LIBRARIES to Ascent base script. CMAKE was picking up python's openblas otherwise.

* Error with unzip.

* Apply pre-commmit fixes

* Add working build on ascent.

* Add working gcc11.2.0 spack spec.

* Add Ascent Spack pipeline. [ascent-rebuild]

* Update gcc version to 11.2.0 in base.sh [skip-ci]

* Fix stages of Ascent pipeline [ascent-rebuild]

* Add working ascent spack build.

* Add hiop@develop force rebuild to PNNL CI [ascent-rebuild] [newell-rebuild] [deception-rebuild] [incline-rebuild].

* Update Ascent spack built tcl modules

* Only test ascent on tcl module update [ci-skip]

* Update base.sh to disable python on ascent [skip ci]

* Remove LAPACK_LIBRARIES spec [ascent-test]

* Update ascent.gitlab-ci.yml to fix needs/dependencies [ascent-test]

* Update deception spack built tcl modules - [deception-test]

* Try again with Python, but have Spack build it instead of using the external module [ascent-rebuild]

* Force python rebuild on ascent and use [email protected] on incline [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Pin [email protected] on all CI platforms [decetpion-rebuild] [ascent-rebuild] [newell-rebuild] [incline-rebuild]

* Fix false positive/negative in Ascent pipelines [deception-rebuild] [ascent-test]

* Update incline spack built tcl modules - [incline-test]

* Update newell spack built tcl modules - [newell-test]

* Fix HiOp spec on Ascent [ascent-rebuild].

* Update deception spack built tcl modules - [deception-test]

* Update CPU Spack build with issue for each failing build [ci skip]

* Update Ascent spack built tcl modules [ascent-test]

* Add 1.0.0 dep into CHANGELOG.

* Add ascent-skip to CI to get tests passing [ascent-test]

---------

Co-authored-by: nkoukpaizan <[email protected]>
Co-authored-by: Cameron Rutherford <[email protected]>
Co-authored-by: cameronrutherford <[email protected]>
Co-authored-by: spack-auto-module <[email protected]>

* Add Spack CPU build with `exago+hiop+raja~ipopt ^hiop+raja~sparse` (#41)

* Add CPU build with hiop+sparse and exago~ipopt+hiop+raja

* Update .github/workflows/spack_cpu_build.yaml

* `+mpi` to `+raja` CPU build

* Add HIOPRAJASPARSE model if sparse and raja enabled

* Fix other HIOPRAJASPARSE ifdef

* pflow functionality tests fully mpi aware

* add logging rank variable

* Apply pre-commmit fixes

* Deleted unused header file.

* Brought SCOPFLOW test driver in line with PFLOW driver.

* Applied additional changes to selfcheck.cpp file for PFLOW, SOPFLOW and SCOPFLOW
to adapt tests for running on multiple MPI ranks.

* Apply pre-commmit fixes

* Apply pre-commmit fixes

* Updated third party libraries

* Set more default values in selfcheck.cpp to get rid of uninitialized variables
errors in Valgrind and modified a few test values so that tests pass.

* Apply pre-commmit fixes

* Fixed up some preprocessor glitches that got introduced in the rebase.

* Modified versions on pybind11 and spack to match develop.

* Fix remaining issues in merge request.

* Apply pre-commmit fixes

* Fixed preprocessor directives to match develop branch.

* Modified constructor of FunctionalityTestContext to get rid of a bunch of code
checking MPI calls.

* Apply pre-commmit fixes

* Remove logging ranks variable.

---------

Co-authored-by: Jaelyn Litzinger <[email protected]>
Co-authored-by: Bruce J Palmer <[email protected]>
Co-authored-by: Nicholson Koukpaizan <[email protected]>
Co-authored-by: Cameron Rutherford <[email protected]>
Co-authored-by: Bill <[email protected]>
Co-authored-by: Abhyankar, Shrirang G <[email protected]>
Co-authored-by: nkoukpaizan <[email protected]>
Co-authored-by: cameronrutherford <[email protected]>
Co-authored-by: spack-auto-module <[email protected]>
Co-authored-by: Bruce J Palmer <[email protected]>
@nkoukpaizan nkoukpaizan deleted the nicholson/upgrade-ascent-buildsystem branch September 24, 2024 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

exago+python fails to build with Spack on Ascent Port Spack module rebuild pipelines to Ascent
4 participants