Singularity Summary and Tips

From Storrs HPC Wiki
Revision as of 17:45, 15 November 2019 by Xil14026 (talk | contribs)
Jump to: navigation, search

Why we need to use singularity in HPC cluster

1: The major reason: some of software or packages are kernel sensitive, for example they require the latest version of glibc, which is not compatible with the host OS in HPC cluster. Therefore, we use singularity to solve this issue. 2: Since our host OS is redhat 6.7, please do NOT use ubuntu with version higher than 16.10 and centos higher than 7.

How to create singularity recipe file

Three important sections should be defined in recipe file.

1: BootStrap: in this section you can figure out use which method (library, docker, debootstrap/yum) to get which OS (ubuntu, centos, redhat) with which version (trusty, xenial) from where (singularity hub, docker hub, ubuntu/centos official website) and install which libraries (bash vim less) by default.

  Example 1 
  BootStrap: debootstrap
  OSVersion: xenial
  Include: bash vim less man-db apt-utils tzdata
  Example 2
  BootStrap: docker
  From: ubuntu:xenial

2: %environment: in this section, you can figure out the language and location of OS. inaddition, you an export some paths into environment variables inside singularity container.

      export LC_ALL LD_LIBRARY_PATH TZ   

3: %post: in this section, you can figure out all customized script that be used to install the specific libraries, software and packages.

      #Get 3DSlicer
      wget -P /home/software/3DSlicer
      tar -xvf /home/software/3DSlicer/1023242 -C /home/software/3DSlicer		
      #For 3DSlicer
      apt-get install -y libfontconfig1 \
                         libxrender1 \
                         libgl1-mesa-dev \
                         libglu1-mesa \
                         libglu1-mesa-dev \
                         libxtst6 \

4: Example

(1)The singularity recipe file for 3DSlicer:
(2)The singularity recipe file for tensorflow with ubuntu xenial, python 3.6.7 and cuda 9.0:

How to create singularity container

1: There are two types of container, including "image" and "sandbox".

2: Any singularity container should be created by using "sudo".

Now, we use the recipe file of 3D Slicer as an example:


sudo singularity build ./3DSlicer.ssb ../singularity_3DSlicer.recipe

1: Singularity image container is a binary file.
2: The size of image container is small.
1: Hard to be expanded (size is fixed).
2: You are not allowed to install new packages and libraries once the image been created.


sudo singularity build --sandbox ./3DSlicer.ssb ../singularity_3DSlicer.recipe

1: It is very easy to be expanded (no size limitation). 
2: You can install any additional packages and libraries after the container been created.
3: Sandbox is very easy to be converted to image by using the command line below:
   sudo singularity build 3DSlicer.sif 3DSlicer.ssb/

1: Singularity sandbox container is a folder.
2: The size of sandbox container is around 3 times larger than image container.

Recommendation: since the storage is not a big issue for modern computer, sandbox is good choice to used when build singularity container.

How to run singularity container in HPC Cluster

Usually, the target