T64 Debian using Gcc

This set of build instructions builds the tpetra branch of Albany in 64 bit mode, in other words with support of a > 2.1 billion word address space to handle problems with very large numbers of degrees of freedom.

This is the avatar server at RPI:

Linux avatar 2.6.32-5-amd64 #1 SMP Mon Feb 29 00:33:51 UTC 2016 x86_64 GNU/Linux

The compilers are gcc 4.8.4, OpenMPI 1.8.3, and Debian linux.

Build supporting packages

Trilinos requires a set of "third party libraries" (TPLs) to support Albany. The actual libraries needed depends on the final Albany configuration desired. These instructions build a superset, all the libraries needed to support any Albany configuration.

Notes: In this example

The TPLs are installed in /users/ghansen

Building ZLib

Version 1.2.8

  cd /fasttmp/ghansen
  wget http://zlib.net/zlib-1.2.8.tar.gz
  tar -xvf zlib-1.2.8.tar.gz
  cd zlib-1.2.8
  mkdir build
  cd build
  cmake \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_C_FLAGS="-O3" \
  -DCMAKE_INSTALL_PREFIX=/users/ghansen \
  ..

  make -j 8
  make install

Building HDF5

Version 1.8.16

  tar -xvf hdf5-1.8.16.tar.gz
  cd hdf5-1.8.16
  ./configure --prefix=/users/ghansen CC=/users/ghansen/ompi-gcc/bin/mpicc \
  CXX=/users/ghansen/ompi-gcc/bin/mpicxx FC=/users/ghansen/ompi-gcc/bin/mpif90 \
  CXXFLAGS="-fPIC -O3 -march=native" CFLAGS="-fPIC -O3 -march=native" FFLAGS="-fPIC -O3 -march=native" \
  --enable-parallel --enable-shared --with-zlib=/users/ghansen
  make -j 4
  make install

Building parallel-netcdf

Version 1.7.0

Configure and build:

  ./configure --prefix=/users/ghansen CC=/users/ghansen/ompi-gcc/bin/mpicc \
  FC=/users/ghansen/ompi-gcc/bin/mpif90 CXX=/users/ghansen/ompi-gcc/bin/mpicxx \
  CXXFLAGS="-O3 -march=native -fPIC" CFLAGS="-O3 -march=native -fPIC" FFLAGS="-O3 -march=native -fPIC"
  make -j 8 
  make install

Building Netcdf

Version 4.4.0

Edit the file include/netcdf.h. Make the following edits near line 265:

  #define NC_MAX_DIMS    65536    /* max dimensions per file */
  #define NC_MAX_ATTRS    8192    
  #define NC_MAX_VARS   524288    /* max variables per file */
  #define NC_MAX_NAME   256 
  #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */

Now, configure and build:

  ./configure --prefix=/users/ghansen CC=/users/ghansen/ompi-gcc/bin/mpicc \
  FC=/users/ghansen/ompi-gcc/bin/mpif90 CXX=/users/ghansen/ompi-gcc/bin/mpicxx \
  CXXFLAGS="-I/users/ghansen/include -O3 -march=native -fPIC" \
  CFLAGS="-I/users/ghansen/include -O3 -march=native -fPIC" LDFLAGS="-L/users/ghansen/lib -O3 -march=native -fPIC" \
  --disable-fsync --enable-shared --disable-doxygen --enable-netcdf-4 --enable-pnetcdf
  make -j 4 
  make install

Building boost

Version 1.60.0

Build boost with the commands

  rm ~/user-config.jam
  echo "using gcc : 4.8.4 : /users/ghansen/bin/g++ ;" >> ~/user-config.jam
  echo "using mpi : /users/ghansen/ompi-gcc/bin/mpicxx ;" >> ~/user-config.jam
  ./bootstrap.sh --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=/users/ghansen
  ./b2 -j 4
  ./b2 -j 4 install

Building ParMetis

Version 4.0.3

  cd /fasttmp/ghansen
  wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
  tar -xvf parmetis-4.0.3.tar.gz
  cd parmetis-4.0.3

Edit metis.h to use an IDXTYPEWIDTH of 64

  cd metis/include
  vi metis.h
  #define IDXTYPEWIDTH 64

Run CMake to build parmetis

  cd /fasttmp/ghansen/parmetis-4.0.3
  mkdir build2
  cd build2
  export metis=/fasttmp/ghansen/parmetis-4.0.3/metis
  cmake \
  -DCMAKE_INSTALL_PREFIX=/users/ghansen \
  -DMETIS_PATH=$metis \
  -DGKLIB_PATH=$metis/GKlib \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  ..
  make -j 4
  make install

  cp libmetis/libmetis.a /users/ghansen/lib
  cp $metis/include/metis.h /users/ghansen/include

SuperLU

Version 4.3

  wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
  tar -xvf superlu_4.3.tar.gz
  cd SuperLU_4.3

Build paths

  mkdir /users/ghansen/SuperLU_4.3
  mkdir /users/ghansen/SuperLU_4.3/include
  mkdir /users/ghansen/SuperLU_4.3/lib

Edit make.inc for your machine and environment

  PLAT = _linux
  SuperLUroot = /users/ghansen/SuperLU_4.3
  SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
  TMGLIB          = libtmglib.a
  BLASDEF     = -DUSE_VENDOR_BLAS
  BLASLIB     = -L/usr/lib -lblas
  LIBS        = $(SUPERLULIB) $(BLASLIB)
  ARCH         = ar
  ARCHFLAGS    = cr
  RANLIB       = ranlib
  CC           = gcc
  CFLAGS       = -O3
  NOOPTS       =
  FORTRAN      = gfortran
  FFLAGS       = -O3
  LOADER       = $(CC)
  LOADOPTS     =
  CDEFS        = -DAdd_
  MATLAB       = /usr/sww/matlab

Build SuperLU

  make -j 4
  cd SRC
  cp *.h /users/ghansen/SuperLU_4.3/include

Build HWLOC

HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

Version 1.9

  wget https://www.open-mpi.org/software/hwloc/v1.9/downloads/hwloc-1.9.tar.gz
  tar -xvf hwloc-1.9.tar.gz
  cd hwloc-1.9
  ./configure CC=mpicc CXX=mpicxx --prefix=/users/ghansen
  make -j 4
  make install

Build Trilinos

do-configure script

  #!/bin/sh
  #
  export TRILINOS_HOME=/lore/ghansen/Trilinos/publicTrilinos
  BUILD_DIR=`pwd`
  INSTALL_DIR=/users/ghansen/Trilinos/MPI_REL
  LIB_DIR=/users/ghansen
  MPI_BASE_DIR=/users/ghansen
  SUPERLUDIR=${LIB_DIR}/SuperLU_4.3
  
  export BOOST_ROOT=$BOOST_DIR
  EXTRA_ARGS=$@
  cmake \
      -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
      -D CMAKE_BUILD_TYPE:STRING=RELEASE \
      -D Trilinos_EXTRA_REPOSITORIES:STRING=SCOREC \
      -D Trilinos_ENABLE_SCOREC:BOOL=ON \
      -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
  	-D Trilinos_ENABLE_SCORECpumi_geom_parasolid:BOOL=ON \
  \
  	-D TPL_ENABLE_Parasolid:BOOL=ON \
  	-D Parasolid_INCLUDE_DIRS:PATH="/usr/local/parasolid/25.1.181" \
  	-D Parasolid_LIBRARY_DIRS:PATH="/usr/local/parasolid/25.1.181/shared_object" \
  \
      -D TPL_ENABLE_MPI:BOOL=ON \
      -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
  \
      -D Trilinos_ENABLE_Kokkos:BOOL=ON \
      -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
      -D Trilinos_ENABLE_KokkosContainers:BOOL=ON \
      -D Trilinos_ENABLE_KokkosLinAlg:BOOL=ON \
      -D Trilinos_ENABLE_KokkosCompat:BOOL=ON \
  \
      -D Boost_INCLUDE_DIRS:PATH=$LIB_DIR/include \
      -D BoostLib_INCLUDE_DIRS:PATH=$LIB_DIR/include \
      -D BoostAlbLib_INCLUDE_DIRS:PATH=$LIB_DIR/include \
      -D Boost_LIBRARY_DIRS:PATH=$LIB_DIR/lib \
      -D BoostLib_LIBRARY_DIRS:PATH=$LIB_DIR/lib \
      -D BoostAlbLib_LIBRARY_DIRS:PATH=$LIB_DIR/lib \
  \
      -D TPL_ENABLE_Netcdf:STRING=ON \
      -D Netcdf_INCLUDE_DIRS:PATH="${LIB_DIR}/include" \
      -D Netcdf_LIBRARY_DIRS:PATH="${LIB_DIR}/lib" \
      -D TPL_ENABLE_HDF5:STRING=ON \
      -D HDF5_INCLUDE_DIRS:PATH="${LIB_DIR}/include" \
      -D TPL_HDF5_LIBRARIES:PATH="${LIB_DIR}/lib/libnetcdf.a;${LIB_DIR}/lib/libhdf5_hl.a;${LIB_DIR}/lib/libhdf5.a;${LIB_DIR}/lib/libz.a" \
      -D TPL_ENABLE_Zlib:STRING=ON \
      -D Zlib_INCLUDE_DIRS:PATH="${LIB_DIR}/include" \
      -D TPL_Zlib_LIBRARIES:PATH="${LIB_DIR}/lib/libz.a" \
  \
      -D TPL_ENABLE_ParMETIS:STRING=ON \
      -D ParMETIS_INCLUDE_DIRS:PATH="${LIB_DIR}/include" \
      -D ParMETIS_LIBRARY_DIRS:PATH="${LIB_DIR}/lib" \
  \
      -D TPL_ENABLE_SuperLU:STRING=ON \
      -D SuperLU_INCLUDE_DIRS:STRING="${SUPERLUDIR}/include" \
      -D SuperLU_LIBRARY_DIRS:STRING="${SUPERLUDIR}/lib" \
  \
      -D Trilinos_ASSERT_MISSING_PACKAGES=OFF \
      -D CMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} \
  \
  $EXTRA_ARGS \
  ${TRILINOS_HOME}

NOTE: If you use the MKL libraries, please make sure you use the 32 bit Intel library mkl_intel_lp64, NOT the 64 bit version mkl_intel_ilp64. Both Lapack and Blas functions are called within Albany's local (on rank) address space, which is the local ordinal (LO) size, typically int.

To build Trilinos, use an out-of-source build. I usually make a build directory that matches the configuration of Trilinos that I am building in the Trilinos root directory:

    mkdir /lore/ghansen/tri_build
	cd /lore/ghansen/tri_build
	mkdir MPI_REL

Place the above do-configure file into the MPI_REL directory, change directories into MPI_REL, and perform the configure, build, and install:
```
  cd MPI_REL
  ./do-configure
  make -j 4
  make -j 4 install
```

Build Albany

CMake configure script:

  #!/bin/sh
  
  export ALB_64BIT_INT=ON
  export ALB_ENABLE_SCOREC=ON
  export ALB_ENABLE_LCM=ON
  export ALB_ENABLE_LAME=ON
  export ALB_ENABLE_HYDRIDE=ON
  export ALB_ENABLE_QCAD=ON
  export ALB_ENABLE_MOR=ON
  export ALB_ENABLE_ASCR=ON
  export ALB_ENABLE_AERAS=ON
  export ALB_ENABLE_FELIX=ON
  export ALB_ENABLE_AlbanyCI=OFF
  export ALB_DEBUG=OFF
  export ALB_ENABLE_CHECK_FPE=OFF
  export ALB_ENABLE_SPECULATIVE=OFF
  export ALB_ENABLE_SG_MP=ON
  export ALB_ENABLE_PERF_TESTS=ON
  export TRILINOS_INSTALL_DIR=/users/ghansen/Trilinos/MPI_REL
  
  export ALB_VERBOSE=OFF
  
  cmake \
        -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
        -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
        -D ENABLE_AERAS:BOOL=${ALB_ENABLE_AERAS} \
        -D ENABLE_QCAD:BOOL=${ALB_ENABLE_QCAD} \
        -D ENABLE_HYDRIDE:BOOL=${ALB_ENABLE_HYDRIDE} \
        -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
        -D ENABLE_LAME:BOOL=${ALB_ENABLE_LAME} \
        -D CMAKE_VERBOSE_MAKEFILE:BOOL=${ALB_VERBOSE} \
        -D ENABLE_DEBUGGING:BOOL=${ALB_DEBUG} \
        -D ENABLE_CHECK_FPE:BOOL=${ALB_ENABLE_CHECK_FPE} \
        -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
        -D ENABLE_FELIX:BOOL=${ALB_ENABLE_FELIX} \
        -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
        -D ENABLE_ALBANY_CI:BOOL=${ALB_ENABLE_AlbanyCI} \
        -D ENABLE_ASCR:BOOL=${ALB_ENABLE_ASCR} \
        -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
        -D ENABLE_PERFORMANCE_TESTS:BOOL=${ALB_ENABLE_PERF_TESTS} \
        -D ENABLE_64BIT_INT:BOOL=${ALB_64BIT_INT} \
        -D ALBANY_CTEST_TIMEOUT:INTEGER=70 \
  \
  ..

Clone the Albany repo in a suitable place:

	cd /users/ghansen
	git clone https://github.com/SNLComputation/Albany.git

Create an out-of-source build directory
```
  cd Albany
  mkdir build
```
Place the above configure script into the build directory, then configure, build, and test
```
  cd build
  ./buildAlbany
  make -j 4
  ctest
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

T64 Debian using Gcc

Build supporting packages

Building ZLib

Building HDF5

Building parallel-netcdf

Building Netcdf

Building boost

Building ParMetis

SuperLU

Build HWLOC

Build Trilinos

Build Albany

Docker Image

SPACK Package

ToDo List

Albany FAQs

Capabilities

Sub-Projects

Compiling

Nightly Tests

Code HowTo

User Group Meetings

PAALS Tutorials

Wiki HowTo

Visualization Utilities

Good Commits

Python and Albany

Clone this wiki locally