Singularity Summary and Tips
- 1 Why we need to use singularity in HPC cluster
- 2 How to create singularity recipe file
- 3 How to create singularity container
- 4 How to run singularity container in HPC Cluster
- 5 Tips
Why we need to use singularity in HPC cluster
1: The major reason: some of software or packages are kernel sensitive, for example they require the latest version of glibc, which is not compatible with the host OS in HPC cluster. Therefore, we use singularity to solve this issue.
2: Since our host OS is redhat 6.7, please do NOT use ubuntu with version higher than 16.10 and centos higher than 7.
3: For the reason why we need to use singularity in HPC cluster rather than docker, please read the link: https://dev.to/grokcode/singularity--a-docker-for-hpc-environments-i6p
How to create singularity recipe file
Three important sections should be defined in recipe file.
1: BootStrap: in this section you can figure out use which method (library, docker, debootstrap/yum) to get which OS (ubuntu, centos, redhat) with which version (trusty, xenial) from where (singularity hub, docker hub, ubuntu/centos official website) and install which libraries (bash vim less) by default.
Example 1 BootStrap: debootstrap OSVersion: xenial MirrorURL: http://us.archive.ubuntu.com/ubuntu/ Include: bash vim less man-db apt-utils tzdata --------- Example 2 BootStrap: docker From: ubuntu:xenial
2: %environment: in this section, you can figure out the language and location of OS. inaddition, you an export some paths into environment variables inside singularity container.
%environment LC_ALL=C LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/usr/local/cuda/extras/CUPTI/lib64 TZ='America/New_York' export LC_ALL LD_LIBRARY_PATH TZ
3: %post: in this section, you can figure out all customized script that be used to install the specific libraries, software and packages.
%post #Get 3DSlicer wget -P /home/software/3DSlicer https://download.slicer.org/bitstream/1023242 tar -xvf /home/software/3DSlicer/1023242 -C /home/software/3DSlicer #For 3DSlicer apt-get install -y libfontconfig1 \ libxrender1 \ libgl1-mesa-dev \ libglu1-mesa \ libglu1-mesa-dev \ libxtst6 \ libegl1-mesa-dev
(1)The singularity recipe file for 3DSlicer: https://github.com/lxwgcool/singularity/blob/master/singularity_3DSlicer.recipe (2)The singularity recipe file for tensorflow with ubuntu xenial, python 3.6.7 and cuda 9.0: https://github.com/lxwgcool/singularity/blob/master/Singularity.tensorflow-gpu-1.12.0
How to create singularity container
1: There are two types of container, including "image" and "sandbox".
2: Any singularity container should be created by using "sudo".
Now, we use the recipe file of 3D Slicer as an example: https://github.com/lxwgcool/singularity/blob/master/singularity_3DSlicer.recipe
sudo singularity build ./3DSlicer.ssb ../singularity_3DSlicer.recipe Advantage 1: Singularity image container is a binary file. 2: The size of image container is small. Disadvantage 1: Hard to be expanded (size is fixed). 2: You are not allowed to install new packages and libraries once the image been created.
sudo singularity build --sandbox ./3DSlicer.ssb ../singularity_3DSlicer.recipe Advantage 1: It is very easy to be expanded (no size limitation). 2: You can install any additional packages and libraries after the container been created. 3: Sandbox is very easy to be converted to image by using the command line below: sudo singularity build 3DSlicer.sif 3DSlicer.ssb/ Disadvantage 1: Singularity sandbox container is a folder. 2: The size of sandbox container is around 3 times larger than image container.
Recommendation: since the storage is not a big issue for modern computer, sandbox is good choice to used when build singularity container.
How to run singularity container in HPC Cluster
1: singularity container path: /apps2/singularity/general/general_python3
2: python code: /home/xil14026admin/Test/tensorflow/Tensorflow_Test.py
CPU Mode (sbatch script)
#!/bin/bash #SBATCH -N 1 #require 1 node #SBATCH -p general #use general partition #SBATCH -n 1 #use 1 cpu module purge module load singularity/3.1 singularity exec /apps2/singularity/general/general_python3/ python3 Tensorflow_Test.py
GPU Mode (sbatch script)
#!/bin/bash #SBATCH -N 1 #require 1 node #SBATCH -p gpu #use general partition #SBATCH -n 1 #use 1 cpu #SBATCH --gres=gpu:1 #use 1 gpu card module purge module load singularity/3.1 cuda/10.0 singularity exec --nv -B /apps2/cuda/10.0/lib64:/home/extraLibs/cuda,/apps2/cudnn/7.4.2/cuda/lib64:/home/extraLibs/cudnn /apps2/singularity/general/general_python3 python3 Tensorflow_Test.py
1: In order to use GPU mode to run the deep learning based framework, make sure use "gres" to apply GPU card in cluster.
- Otherwise, your program will run in CPU mode.
2: In order to run GPU based program successfully, you need to introduce three libraries from cluster into singularity container, including:
- Nvidia driver library
- Cuda library
- Cudnn library
3: How to mount these three libraries from HPC cluster into singularity container:
- Nvidia driver: --nv
- Cuda Library and CUDNN library: we need to manually bind them by sing "-B"
- The environmental variable (LD_LIBRARY_PATH) should be manually set inside singularity container.
singularity exec --nv -B /apps2/cuda/10.0/lib64:/home/extraLibs/cuda,/apps2/cudnn/7.4.2/cuda/lib64:/home/extraLibs/cudnn /apps2/singularity/general/general_python3 python3 Tensorflow_Test.py