Difference between revisions of "Compiling software"

From Storrs HPC Wiki
Jump to: navigation, search
(Explain how RPATH works)
(Highlight rpath output by readelf)
Line 292: Line 292:
We can force the program to use the newer <code>/apps2</code> version of libcurl using the <code>-Wl,-rpath, ...</code>
We can force the program to use the newer <code>/apps2</code> version of libcurl using the <code>-Wl,-rpath, ...</code>
<syntaxhighlight lang="bash" highlight="6">
<syntaxhighlight lang="bash" highlight="6,12">
LDFLAGS="-l curl -Wl,-rpath,/apps2/libcurl/7.60.0/lib" make -B ver
LDFLAGS="-l curl -Wl,-rpath,/apps2/libcurl/7.60.0/lib" make -B ver
# cc  -l curl -Wl,-rpath,/apps2/libcurl/7.60.0/lib  ver.c  -o ver
# cc  -l curl -Wl,-rpath,/apps2/libcurl/7.60.0/lib  ver.c  -o ver

Revision as of 14:48, 8 January 2020

Compiling allows you to access to the latest and greatest software. If you have never compiled software before, the process may seem a little involved at first. This guide will help you understand how it all works.

First, we use short code snippets to understand a few concepts. Then we will see more elaborate examples.



The PATH controls where the shell searches for programs to run. On the cluster, we frequently change the PATH to run programs we installed ourselves.

Let's get an appreciation for how the PATH works with a short exercise of creating a program and running it.

Inside of your shell on the cluster, try to run the command hello

# Run our first program
# -bash: hello: command not found

The above message tells us there is no program named hello in any of the usual places. So then the question is what are the usual places? The command which tells us the locations of the usual places it searches for programs:

# Where does the computer search for programs to run?
which hello
# /usr/bin/which: no hello in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/gpfs/gpfs1/slurm/misc/stubl/stubl-master/bin)

You can see a list of different directories separated by a colon.

This list of directories is nothing but the PATH variable:

# The "which" program searches for programs in the directories stored in the variable PATH.
echo $PATH
# /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/gpfs/gpfs1/slurm/misc/stubl/stubl-master/bin
# Programs like "cat" and "gcc" are stored in /bin and /usr/bin respectively.
which cat
# /bin/cat
which gcc
# /usr/bin/gcc

A PATH is a particular type of variable called an "environmental" variable. An environmental variable is stored in the shell, and is therefore available to any program run from the shell, like which.

Let's create a program hello:

# Create the directory to store our programs.
mkdir -p ~/.apps/hello
# Create the program hello
echo '#!/bin/bash' > ~/.apps/hello/hello
echo 'echo "Hi, there!"' >> ~/.apps/hello/hello
# Give the program executable permission.
chmod +x ~/.apps/hello/hello
# Run hello.
# Hi, there!

But we can do better! Add hello to the PATH so that we don't have to type the directory ~/.apps/hello/ everytime we want to run <hello>

# Tell the shell to first look in ~/.apps/hello before other places.
# Run hello.
# Hi, there!


  • Learned how the PATH variable tells the shell where to find sofware to run.
  • Created our own program called hello.
  • Learned how to add our program to the PATH so that one does not have to remember where it is to located and simply run it by name.


To avoid reinventing the wheel, nearly all scientific programs re-use code from shared "libraries". These names of these shared library files end with the extension .so

However when computer code re-uses these library programs, we sometimes need to tell the compiler the name of the library to use. We will see how to do this in the next section on LDFLAGS.


Now we are ready to compile our first program.

The program blow will inspect the high speed InfiniBand network port present on all the nodes.

LDFLAGS are a variable that tells the compiler

#include <stdio.h>

#include <infiniband/arch.h>
#include <infiniband/verbs.h>

int main() {
  int i, num_devices;
  struct ibv_device **dev_list = ibv_get_device_list(&num_devices);
  printf("Infiniband Devices:\n");
  if (! dev_list)
    printf("None found");
  else {
    for (i = 0; i < num_devices; ++i)
      printf("%-16s\n", ibv_get_device_name(dev_list[i]));
  return 0;

However the program needs to know the name of the InfiniBand library to use. If we neglect to mention the name of the library and try to compile the file directly, it complains about the missing function references:

make ib
cc     ib.c   -o ib
/tmp/ccKfTgaM.o: In function `main':
ib.c:(.text+0x10): undefined reference to `ibv_get_device_list'
ib.c:(.text+0x5a): undefined reference to `ibv_get_device_name'
ib.c:(.text+0x7a): undefined reference to `ibv_free_device_list'
collect2: ld returned 1 exit status
make: *** [ib] Error 1

We need to tell the compiler to use the ibverbs library:

LDFLAGS="-l ibverbs" make ib
# cc   -l ibverbs  ib.c   -o ib
# Infiniband Devices:
# mlx4_0

The -l flag tells the compiler that the next word following it ibverbs is the name of the shared library it should use to create the final program we want, ib.

For those of you familiar with writing code for compiling programs, you might be surprised to see the use of make without any input Makefile. The reason we can skip having any input Makefile is because of the automatic rules feature of make; namely make knows how to compile C programs without us explaining the variable name substitutions to use.


Header files are different than shared libraries, in that they are only needed during compilation time and never again at runtime. Whereas library names end with .so, header file names end with .h for C code (or .hpp for C++ code).

Sometimes header files store information in a manner similar to libraries, in that they stored common useful code instead of simply outlining what is contained in the library; some libraries call themselves "header-only libraries" for this reason which in practise means that those libraries don't have any associated .so files.


The concept of CPATH is similar to PATH, except instead of executable programs, it controls where the shell searches for header files (also called headers) for programs to use.

This example below prints the OpenSSL library version using the OPENSSL_VERSION_TEXT symbol present in the opensslv.h header file:

#include <stdio.h>

#include <openssl/opensslv.h>

int main() {
  printf("%s\n", OPENSSL_VERSION_TEXT);
  return 0;

The system version of OpenSSL is 1.0.1e:

make ver
# cc     ver.c   -o ver
# OpenSSL 1.0.1e 11 Feb 2013

To use a more recent version of OpenSSL 1.0.2o, we must tell the compiler where to look by setting the CPATH variable:

CPATH=/apps2/openssl/1.0.2o/include make -B ver
# cc     ver.c   -o ver
# OpenSSL 1.0.2o  27 Mar 2018

Using module automatically sets the CPATH variable for us so that we don't have to worry about it:

module purge
module load zlib/1.2.11 openssl/1.0.2o
make -B ver
# cc     ver.c   -o ver
# OpenSSL 1.0.2o  27 Mar 2018

We can use the "show" command to see how CPATH is being set:

module show openssl/1.0.2o

setenv           MOD_APP openssl 
setenv           MOD_VER 1.0.2o 
module           load pre-module 
prereq   zlib/1.2.11 
conflict         openssl 
prepend-path     PATH /apps2/openssl/1.0.2o/bin 
prepend-path     LD_LIBRARY_PATH /apps2/openssl/1.0.2o/lib 
prepend-path     LIBRARY_PATH /apps2/openssl/1.0.2o/lib 
prepend-path     LD_RUN_PATH /apps2/openssl/1.0.2o/lib 
prepend-path     INCLUDE /apps2/openssl/1.0.2o/include 
prepend-path     CPATH /apps2/openssl/1.0.2o/include 
prepend-path     MANPATH /apps2/openssl/1.0.2o/ssl/man 
prepend-path     PKG_CONFIG_PATH /apps2/openssl/1.0.2o/lib/pkgconfig 
module           load post-module 


The concept of LD_LIBRARY_PATH is similar to PATH, except instead of executable programs, it controls where the shell searches for libraries for programs to use.

In many cases, we use LD_LIBRARY_PATH together with RPATH; we will learn more about RPATH in the next section.


Using RPATH tells the compiler to modify the final library or executable program that it creates with a library search path to use at runtime.

In other words, it encodes LD_LIBRARY_PATH directly into the executable itself so that LD_LIBRARY_PATH is no longer needed.

You must use RPATHs whenever you're trying to take precedence over a system library.

Consider how using the libcurl.so library located at /usr/lib/libcurl.so always takes precendece over /apps2/libcurl/7.60.0/lib:

#include <stdio.h>

#include <curl/curl.h>

int main() {
  printf("%s\n", curl_version());
  return 0;
LDFLAGS="-l curl" make -B ver
# cc   -l curl  ver.c   -o ver
# libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
ldd ver | grep curl
#         libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x0000003995c00000)

We can force the program to use the newer /apps2 version of libcurl using the -Wl,-rpath, ...

LDFLAGS="-l curl -Wl,-rpath,/apps2/libcurl/7.60.0/lib" make -B ver
# cc   -l curl -Wl,-rpath,/apps2/libcurl/7.60.0/lib  ver.c   -o ver
# libcurl/7.60.0 OpenSSL/1.0.2o zlib/1.2.11
ldd ver | grep curl
#         libcurl.so.4 => /apps2/libcurl/7.60.0/lib/libcurl.so.4 (0x00002b1aeba4a000)
readelf -d ver | head -6
# Dynamic section at offset 0x7d0 contains 22 entries:
#   Tag        Type                         Name/Value
#  0x0000000000000001 (NEEDED)             Shared library: [libcurl.so.4]
#  0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
#  0x000000000000000f (RPATH)              Library rpath: [/apps2/libcurl/7.60.0/lib]

Another example of needing to force precedence over system libraries is when using modern compilers:

#include <any>                  // Requires >= C++17.
#include <iostream>

int main() {
  std::any value = 1;
  std::cout << value.type().name() << ": " << std::any_cast<int>(value) << std::endl;
  value = 'A';
  std::cout << value.type().name() << ": " << std::any_cast<char>(value) << std::endl;
  return 0;

Here the program compiles fine, but crashes at runtime because it's trying to use the older system version:

module purge
module load gcc/9.2.0
CXXFLAGS=-std=c++17 make test
# ./test: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ./test) 
ldd test
# ./test: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ./test)
#         linux-vdso.so.1 =>  (0x00007ffe97f9f000)
#         libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000037ade00000)
#         libm.so.6 => /lib64/libm.so.6 (0x00000037a9200000)
#         libgcc_s.so.1 => /apps2/gcc/9.2.0/lib64/libgcc_s.so.1 (0x00002b235a6c3000)
#         libc.so.6 => /lib64/libc.so.6 (0x00000037a8600000)
#         /lib64/ld-linux-x86-64.so.2 (0x00000037a8200000)

Use the RPATH setting so that the executable itself knows to use the location of the newer C++ standard library:

CXXFLAGS=-std=c++17 LDFLAGS=-Wl,-rpath,/apps2/gcc/9.2.0/lib64 make test
# 1
# A
ldd test
# ./test: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ./test)
#         linux-vdso.so.1 =>  (0x00007ffd535c4000)
#         libstdc++.so.6 => /apps2/gcc/9.2.0/lib64/libstdc++.so.6 (0x00002b3fe4b99000)
#         libm.so.6 => /lib64/libm.so.6 (0x00000037a9200000)
#         libgcc_s.so.1 => /apps2/gcc/9.2.0/lib64/libgcc_s.so.1 (0x00002b3fe4fa2000)
#         libc.so.6 => /lib64/libc.so.6 (0x00000037a8600000)
#         /lib64/ld-linux-x86-64.so.2 (0x00000037a8200000)

Compiling a large program

Don't I need sudo permissions?

No, using the administrative program sudo is commonly suggested to install software using commands like sudo make install, but sudo is only needed because the default install locations like /usr/local are protected.

As long as you choose a different install location where you have write access, such as a location in your home directory, you don't need any special permissions or sudo.

The setting to change the install location is typically called a "prefix". We will explain how to set the prefix location.

Tarballs of source code

Often source code for GNU/Linux will be provided in a "tarball" file. You can recognize a tarball file by it's file extension; some examples are:

.tar.gz   .tgz    # These 2 are equivalent file extensions
.tar.bz2  .tbz2

You can unpack these files in a directory using tar -xf ${NAME_OF_TARBALL}.tar.gz.

At other times, instead of a tarball, one may need to grab a copy from a version control system like git. In the case of git, one might create the source directory by cloning the source URL.

Now that we have our source files in a directory, the next thing we need to do is consider the compiler to use.

Which compiler should I use?

Usually the developer will suggest which compiler(s) are supported in the documentation. If not, using gcc is safest. Our RedHat 6.7 compute nodes use gcc 4.4.7 by default. If your compilation complains about needing a newer version you can load any of the gcc modules.

Some of our users report better performance with Intel MPI. One can access them from the intelics modules, where the version is the year.

Another option is the Portland Group (PGI) compiler. Intel and PGI tend to be more popular with Fortran programs as are quicker to implement the latest Fortran standards.

Finally, if you are compiling for GPU, you would need to load the nvidia compiler available in the cuda module.

# List all compilers: GNU, Intel, Portland Group, and Nvidia
for compiler in gcc intel pgi cuda ; do module avail $compiler ; done
# Output from the above command
----------------------- /apps2/Modules/3.2.6/modulefiles -----------------------
gcc/4.8.2     gcc/4.9.3     gcc/5.4.0-alt gcc/9.1.0
gcc/4.8.5     gcc/5.4.0     gcc/6.3.0     gcc/9.2.0

----------------------- /apps2/Modules/3.2.6/modulefiles -----------------------
intel/2019u3                      intelics/2016.1-full-gcc
intelics/2012.0.032               intelics/2016.3-full
intelics/2013.1.039-compiler      intelics/2017
intelics/2013.1.039-full(default) intelics/ifort/11.0.084

----------------------- /apps2/Modules/3.2.6/modulefiles -----------------------
pgi/14.6  pgi/15.1  pgi/15.10 pgi/15.7  pgi/16.1  pgi/16.10

----------------------- /apps2/Modules/3.2.6/modulefiles -----------------------
cuda/10.0   cuda/7.0    cuda/8.0    cuda/9.1
cuda/10.1   cuda/7.5    cuda/8.0.61

Before compiling programs, you may want to remove any other modules you have loaded so that they do not interfere with your compilation.

# Unload all modules
module purge

General workflow

Follow the documentation in your software source directory. Typically the workflow is:

./configure --prefix=${HOME}/apps
make -j $(nproc)
make install

Good practise is to create a shell script which runs these commands for you, so that a few months from now you remember exactly how you compiled your software and make your work more reproducible for yourself, your lab mates and collaborators. Also, you may want to write the line set -e toward the top of your shell script so that the script stops when it encounters errors.

Nearly all software that needs compilation will at least ship with a makefile. If you are not an expert in using Makefiles, you should really, really make yourself literate in being able to read and understand them by spending 2 hours reading a short introduction to makefiles such as the Software Carpentry automation and make lesson. The make command will search for a file named Makefile. If one does not exist you would need to specify a file name using e.g. make -f ${NAME_OF_MAKEFILE}.mk.

If your software is complex enough to also require other dependencies, it would likely come with a configure shell script. It is a good idea to run ./configure --help to see how to change variables and set PATHs to libraries. You almost always would need to set the --prefix option to set the final installation path as you do not have access to the system protected directories of /bin /lib64 /usr/local etc. If you obtained your code from version control instead of a traditional release and do not see a configure script and your documentation tells you that you need one, you may likely need to also generate the configure shell script from configure.ac using a program named similar to bootstrap.sh, autogen.sh or at worst you would need to run autoreconf directly.

Good resources for understanding how the autotools programs work that process configure.ac and Makefile.am files are the basics of autotools in the Gentoo Linux development manual, and the Diego Pattenò's comprehensive online book autotools.io


VASP 5.3.3

Reading the comments in the VASP makefiles, we can compile VASP with the PGI compiler or the Intel compiler. As the makefile comments mention there is no performance change with the PGI compiler versions, we will use the Intel compiler in this example:

# Create a directory for our VASP project.
mkdir ~/src/vasp-5.3.3
cd ~/src/vasp-5.3.3

# Copy the source code from the admin directory.
cp -arv /shared/admin/sw-src/rhel6/vasp/vasp.5.3.3.tar.gz .
cp -arv /shared/admin/sw-src/vasp/vasp.5.lib.tar.gz .
# Unpack the sources.
tar -xvpf vasp.5.3.3.tar.gz
tar -xvpf vasp.5.lib.tar.gz

# Load the Intel compiler.
# List available Intel compiler versions:
module avail intelics
# Get rid of any other modules that might interfere with our compilation.
module purge
# The latest version at this time is 2017.
module load intelics/2017

# Compile the VASP 5 library.
cd vasp.5.lib/
# There are several makefiles to compile VASP for different types of CPUs and compilers.
# These are the linux compatible makefiles:
ls -1 makefile.linux*
# The best supported for our cluster is makefile.linux_ifc_P4
# Let's see what the makefile will do before running it.
make -n -f makefile.linux_ifc_P4
# Now compile by running make without the `-n` flag.
# Also overwrite Intel's old Fortran compiler name from `ifc` to be `ifort` by passing as a variable to `make`.
make -f makefile.linux_ifc_P4 FC=ifort
# Go back to src directory
cd ..

# Compile the VASP program.
cd vasp.5.3/
# Compile by running make without the `-n` flag.
# With VASP we cannot use `-j` for simultaneous compilation as it is unreliable.
# Overwrite BLAS variable as Intel now calls the "guide" library as "iomp5" per https://software.intel.com/en-us/forums/intel-c-compiler/topic/284445
make -f makefile.linux_ifc_P4 BLAS=-liomp5\ -mkl

Create a module file for VASP that so that we can conveniently load VASP and it's dependencies. The name that you choose for your module file is important as that is what module uses to reference it. We will make our name different by adding the "-mine" suffix to help separate it from the system installed vasp.

mkdir -p ~/mod/vasp
cd ~/mod/vasp
nano 5.3.3-mine
 1 #%Module1.0
 3 # Throw an error if any of these modules are loaded.
 4 conflict vasp
 5 conflict intelics
 7 # Load the particular Intel compiler module we used for the Math Kernel library, etc.
 8 module load intelics/2017
10 # Modify the PATH to use our compiled VASP.  Do not use a trailing slash.
11 prepend-path PATH ~/src/vasp-5.3.3/vasp.5.3

If you are interested, in learning about module files you can read man modulefile

Finally, make sure that module knows to look in your ~/mod directory for your module files by setting the MODULEPATH environmental variable:

nano ~/.bashrc  # Add the lines below.
1 # My modules
2 source /etc/profile.d/modules.sh

Reload your ~/.bashrc file in your current shell:

1 source ~/.bashrc
2 # Finally Now we can load and run our VASP module
3 module load vasp/5.3.3-mine
4 which vasp
5 vasp -h

GEMMA 0.96

To compile and install your copy of GEMMA, let's fetch the source tarball from GitHub. Looking at the GEMMA releases page on GitHub let's copy the link to the .tar.gz source code. Below is a short installation script:

cd  # Go to the home directory
wget -O GEMMA-0.96.tar.gz https://github.com/xiangzhou/GEMMA/archive/v0.96.tar.gz
tar -xf GEMMA-0.96.tar.gz
cd GEMMA-0.96
cat README.txt  # We need the GSL and LAPACK libraries to compile.

When you try to compile GEMMA now with make all you will get a C++ error:

make all
# g++ -Wall -Weffc++ -O3 -std=gnu++11 -DWITH_LAPACK -m64 -static  -c src/main.cpp -o src/main.o
# cc1plus: error: unrecognized command line option "-std=gnu++11"
# make: *** [src/main.o] Error 1

This is because our GCC 4.4.7 compiler is too old to support the gnu++11 standard. Let's load a newer GCC module, as well as the GSL and LAPACK modules suggested by README.txt:

module load gcc/5.4.0-alt gsl/2.4 lapack/3.5.0 zlib/1.2.8
make -j $(nproc) all FORCE_DYNAMIC=1

We have quite a few module dependencies for GEMMA to run. Let's create a module file to simplify using gemma.

# Create the module file
mkdir -p mod
cat > ~/mod/gemma <<EOF
conflict gemma
module load gcc/5.4.0-alt gsl/2.4 lapack/3.5.0 zlib/1.2.8
prepend-path PATH /home/$USER/GEMMA-0.96/bin

# Add ~/mod to our MODULEPATH for us to be able to load the new gemma module
sed -i '/    module /i \                                               
    export MODULEPATH=$HOME/mod:$MODULEPATH' ~/.bashrc
source ~/.bashrc

# Load the new gemma module
module load gemma
gemma -h