Revision as of 19:22, 15 November 2019

Why we need to use singularity in HPC cluster

1: The major reason: some of software or packages are kernel sensitive, for example they require the latest version of glibc, which is not compatible with the host OS in HPC cluster. Therefore, we use singularity to solve this issue.

2: Since our host OS is redhat 6.7, please do NOT use ubuntu with version higher than 16.10 and centos higher than 7.

3: For the reason why we need to use singularity in HPC cluster rather than docker, please read the link: https://dev.to/grokcode/singularity--a-docker-for-hpc-environments-i6p

How to create singularity recipe file

Three important sections should be defined in recipe file.

1: BootStrap: in this section you can figure out use which method (library, docker, debootstrap/yum) to get which OS (ubuntu, centos, redhat) with which version (trusty, xenial) from where (singularity hub, docker hub, ubuntu/centos official website) and install which libraries (bash vim less) by default.

  Example 1 
  BootStrap: debootstrap
  OSVersion: xenial
  MirrorURL: http://us.archive.ubuntu.com/ubuntu/
  Include: bash vim less man-db apt-utils tzdata
  ---------
  Example 2
  BootStrap: docker
  From: ubuntu:xenial

2: %environment: in this section, you can figure out the language and location of OS. inaddition, you an export some paths into environment variables inside singularity container.

  %environment
      LC_ALL=C
      LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/usr/local/cuda/extras/CUPTI/lib64
      TZ='America/New_York'
      export LC_ALL LD_LIBRARY_PATH TZ   

3: %post: in this section, you can figure out all customized script that be used to install the specific libraries, software and packages.

  %post	
      #Get 3DSlicer
      wget -P /home/software/3DSlicer https://download.slicer.org/bitstream/1023242
      tar -xvf /home/software/3DSlicer/1023242 -C /home/software/3DSlicer		
      #For 3DSlicer
      apt-get install -y libfontconfig1 \
                         libxrender1 \
                         libgl1-mesa-dev \
                         libglu1-mesa \
                         libglu1-mesa-dev \
                         libxtst6 \
                         libegl1-mesa-dev

4: Example

(1)The singularity recipe file for 3DSlicer: https://github.com/lxwgcool/singularity/blob/master/singularity_3DSlicer.recipe
(2)The singularity recipe file for tensorflow with ubuntu xenial, python 3.6.7 and cuda 9.0: https://github.com/lxwgcool/singularity/blob/master/Singularity.tensorflow-gpu-1.12.0

How to create singularity container

1: There are two types of container, including "image" and "sandbox".

2: Any singularity container should be created by using "sudo".

Now, we use the recipe file of 3D Slicer as an example: https://github.com/lxwgcool/singularity/blob/master/singularity_3DSlicer.recipe

Image

sudo singularity build ./3DSlicer.ssb ../singularity_3DSlicer.recipe

Advantage
1: Singularity image container is a binary file.
2: The size of image container is small.
 
Disadvantage
1: Hard to be expanded (size is fixed).
2: You are not allowed to install new packages and libraries once the image been created.

Sandbox

sudo singularity build --sandbox ./3DSlicer.ssb ../singularity_3DSlicer.recipe

Advantage
1: It is very easy to be expanded (no size limitation). 
2: You can install any additional packages and libraries after the container been created.
3: Sandbox is very easy to be converted to image by using the command line below:
   sudo singularity build 3DSlicer.sif 3DSlicer.ssb/

Disadvantage
1: Singularity sandbox container is a folder.
2: The size of sandbox container is around 3 times larger than image container.

Recommendation: since the storage is not a big issue for modern computer, sandbox is good choice to used when build singularity container.

How to run singularity container in HPC Cluster

1: singularity container path: /apps2/singularity/general/general_python3

2: python code: /home/xil14026admin/Test/tensorflow/Tensorflow_Test.py

CPU Mode (sbatch script)

#!/bin/bash
#SBATCH -N 1       #require 1 node
#SBATCH -p general #use general partition
#SBATCH -n 1       #use 1 cpu

module purge
module load singularity/3.1
singularity exec /apps2/singularity/general/general_python3/ python3 Tensorflow_Test.py

GPU Mode (sbatch script)

#!/bin/bash
#SBATCH -N 1         #require 1 node
#SBATCH -p gpu       #use general partition
#SBATCH -n 1         #use 1 cpu
#SBATCH --gres=gpu:1 #use 1 gpu card

module purge
module load singularity/3.1 cuda/10.0
singularity exec --nv -B /apps2/cuda/10.0/lib64:/home/extraLibs/cuda,/apps2/cudnn/7.4.2/cuda/lib64:/home/extraLibs/cudnn /apps2/singularity/general/general_python3 python3 Tensorflow_Test.py

Tips

How to edit environment variable inside singularity container

sbatch script

1: In order to use GPU mode to run the deep learning based framework, make sure use "gres" to apply GPU card in cluster.

• Otherwise, your program will run in CPU mode.

2: In order to run GPU based program successfully, you need to introduce three libraries from cluster into singularity container, including:

• Nvidia driver library
• Cuda library
• Cudnn library

3: How to mount these three libraries from HPC cluster into singularity container:

• Nvidia driver: --nv
• Cuda Library and CUDNN library: we need to manually bind them by sing "-B"
• The environmental variable (LD_LIBRARY_PATH) should be manually set inside singularity container.

4: Example:

singularity exec --nv -B /apps2/cuda/10.0/lib64:/home/extraLibs/cuda,/apps2/cudnn/7.4.2/cuda/lib64:/home/extraLibs/cudnn /apps2/singularity/general/general_python3 python3 Tensorflow_Test.py

Missing library

Printing delay