Page History

...

Rebuild the program with:

The default GNU GCC compiler.
The default Classic Intel compiler.
The default LLVM-based Intel compiler.
The default AMD AOCC.

Use the following command to test and show what versions of the libraries are being used at any point:

No Format
make clean ldd test

Expand

title	Solution

You can perform this test with the following one-liner, exploiting the prgenv module:

No Format
for pe in gnu intel intel-llvm amd; do ml prgenv/$pe; make clean ldd test; echo "******************"; done

Pay attention to the following aspects:

The Lmod module command informs you that it has reloaded the corresponding modules when changing the prgenv. This ensures the libraries used in your program are built with the same compiler for maximum compatibility.
The compiler command changes automatically, since we are using the environment variable $CC in the Makefile.
The include and library flags in the compilation lines are adapted automatically based on the libraries loaded.
The final binary is linked with the corresponding libraries for the version of the compiler as shown by ldd output.

Real-world example: CDO

To put into practice what we have learned so far, let's try to build and install CDO. You would typically not need to build this particular application, since it is already available as part of the standard software stack via modules or easily installable with conda. However, it is a good illustration of how to build a real-world software with dependencies to other software packages and libraries.

The goal of this exercise is for you to be able to build CDO and install it under one of your storage spaces (HOME or PERM), and then successfully run:

No Format
<PREFIX>/bin/cdo -V

You will need to:

Familiarise yourself with the installation instructions of this package in the official documentation.
Decide your installation path and your build path.
Download the source code from the CDO website.
Set up your build environment (i.e. modules, environment variables) for a successful build
Build and install the software
Test that works with the command above.

Make sure that CDO is built at least with support to:

NetCDF
HDF5
SZLIB (hint: use AEC)
ecCodes
PROJ
CMOR
UDUNITS

Tip

title	Building in Batch

It is strongly recommended you bundle all your build process in a job script that you can submit in batch requesting additional cpus so that you can exploit build parallelism with make -j

If you would like a starting point for such a job, you can start from the following example, adding and amending the necessary bits as needed:

Code Block

language	bash
title	build_cdo.sh
collapse	true

#!/bin/bash
#SBATCH -J build_cdo
#SBATCH -o %x.out
#SBATCH -n 8

set -x
set -e
set -u
set -o pipefail

# Get the URL and VERSION for the latest CDO available
URL=$(curl -s https://code.mpimet.mpg.de/projects/cdo/files | grep attachments/download  | sed -e "s:.*href=\"\(.*\)\".*:https\://code.mpimet.mpg.de\1:" | head -n 1)
VERSION=$(echo $URL | sed -e "s:.*/cdo-\(.*\).tar.gz:\1:")

# TODO: Define installation prefix and Build directory
# Hint: Use somewhere in your PERM for installation.
PREFIX=
BUILDDIR=

# Move to our BUILD DIRECTORY
mkdir -p $BUILDDIR
cd $BUILDDIR

# Download source
[ -f cdo-$VERSION.tar.gz ] || wget $URL
[ -d cdo-$VERSION ] || tar xvf cdo-$VERSION.tar.gz
cd cdo-$VERSION

# TODO: Define the environment for the build 


# TODO: Configure the build


# Build
make -j $SLURM_NTASKS

# Install
make install

# Check installed binary
$PREFIX/bin/cdo -V

Expand

title	Solution

This is the complete job script to

Code Block

language	bash
title	build_cdo.sh
collapse	true

#!/bin/bash
#SBATCH -J build_cdo
#SBATCH -o %x.out
#SBATCH -n 8

set -x
set -e
set -u
set -o pipefail

# Get the URL and VERSION for the latest CDO available
URL=$(curl -s https://code.mpimet.mpg.de/projects/cdo/files | grep attachments/download  | sed -e "s:.*href=\"\(.*\)\".*:https\://code.mpimet.mpg.de\1:" | head -n 1)
VERSION=$(echo $URL | sed -e "s:.*/cdo-\(.*\).tar.gz:\1:")

# Define installation prefix and Build directory
PREFIX=$PERM/apps/cdo/$VERSION
#BUILDDIR=$TMPDIR
BUILDDIR=$PERM/apps/cdo/build

# Move to our BUILD DIRECTORY
mkdir -p $BUILDDIR
cd $BUILDDIR

# Download source
[ -f cdo-$VERSION.tar.gz ] || wget $URL
[ -d cdo-$VERSION ] || tar xvf cdo-$VERSION.tar.gz
cd cdo-$VERSION

# Define the environment for the build 
module load aec hdf5 netcdf4 udunits proj cmor ecmwf-toolbox

# We will need to explicitly set rpath for proj and eccodes since CDO build system will not
export LDFLAGS="-Wl,-rpath,$proj_DIR/lib -Wl,-rpath,$ECCODES_DIR/lib"

# Configure the build
./configure --prefix=$PREFIX --with-eccodes=$ECCODES_DIR --with-hdf5=$HDF5_DIR --with-netcdf=$NETCDF4_DIR --with-szlib=$AEC_DIR --with-proj=$proj_DIR --with-cmor=$CMOR_DIR --with-udunits2=$UDUNITS_DIR

# Build
make -j $SLURM_NTASKS

# Install
make install

# Check installed binary
$PREFIX/bin/cdo -V

You can submit it to the batch system with

No Format
sbatch build_cdo.sh

While it builds, you may want to keep an eye on the progress with:

No Format
tail -f build_cdo.out

Make sure the job completes successfully and that the output of the CDO executable you built is what you would expect.

Space shortcuts

Page tree

Versions Compared

Old Version 2

New Version 3

Key

Real-world example: CDO