Difference between revisions of "Singularity Guide"

From Storrs HPC Wiki
Jump to: navigation, search
(Fix CentOS VM setup instructions)
(Add R / rstan example and a few grammatical fixes.)
Line 9: Line 9:
 
}}
 
}}
  
<!-- See https://wiki.hpc.uconn.edu/index.php/Template:Ambox -->
+
<!-- See https://wiki.hpc.uconn.edu/index.php/Template:Ambox
 
{{Ambox
 
{{Ambox
 
| type=content
 
| type=content
 
| issue=This article is a work in progress.
 
| issue=This article is a work in progress.
 
}}
 
}}
 +
-->
  
 
Singularity allows complex software to be run on the cluster,
 
Singularity allows complex software to be run on the cluster,
Line 20: Line 21:
 
you can generate a single "image" file
 
you can generate a single "image" file
 
in which you install all your software,
 
in which you install all your software,
and then you can copy that single image file and run it on the cluster.
+
and then you can copy that single image file to the cluster and run it there.
 
Additionally, singularity allows more directly comparisons of running software between different clusters,
 
Additionally, singularity allows more directly comparisons of running software between different clusters,
 
because you can use the exact same versions of software that may not be installed on the different clusters.
 
because you can use the exact same versions of software that may not be installed on the different clusters.
Line 29: Line 30:
 
* Your computer with macOS, Windows, or a GNU/Linux operating system.
 
* Your computer with macOS, Windows, or a GNU/Linux operating system.
 
* [https://www.virtualbox.org/ VirtualBox] to create a virtual machine in which we will install singularity.
 
* [https://www.virtualbox.org/ VirtualBox] to create a virtual machine in which we will install singularity.
* [https://www.vagrantup.com/ Vagrant]( version 1.9 or later) to greatly simplify administration of our virtual machine.
+
* [https://www.vagrantup.com/ Vagrant] (version 1.9 or later) to greatly simplify administration of our virtual machine.
 
* At least 40 GB of disk space.
 
* At least 40 GB of disk space.
  
Line 35: Line 36:
  
 
To be able to create and edit a singularity image,
 
To be able to create and edit a singularity image,
you root administrator access on a machine.
+
you must have root administrator access on a machine.
 
Therefore you cannot manage singularity containers directly on the cluster;
 
Therefore you cannot manage singularity containers directly on the cluster;
you must install your own copy of singularity on your own computer.
+
you must install your own copy of singularity on your computer.
Singularity makes use of the Linux kernel to run the container,
+
Singularity uses the Linux kernel to run the container,
therefore you need a copy of GNU/Linux installed for which
+
therefore you need to use a GNU/Linux machine for which
 
we will use a virtual machine.
 
we will use a virtual machine.
 
For best results,
 
For best results,
it also helps to use a similar Linux distribution to the environment we will finally be our singularity image.
+
it also helps to use a Linux distribution similar to the environment in which we will finally run our singularity image.
 
As our cluster uses RHEL 6.7,
 
As our cluster uses RHEL 6.7,
 
we will create our singularity image using CentOS
 
we will create our singularity image using CentOS
which for our purposes is functionally identical to RHEL.
+
which, for our purposes, is functionally identical to RHEL.
  
 
If you are using macOS,
 
If you are using macOS,
Line 78: Line 79:
 
cd vm-singularity
 
cd vm-singularity
 
vagrant init centos/7
 
vagrant init centos/7
vagrant box add --provider virtualbox centos/7
+
vagrant box add --provider virtualbox centos/7 # If this fails, you might need to upgrade vagrant
  
 
# Build and start the Vagrant hosted VM
 
# Build and start the Vagrant hosted VM
Line 90: Line 91:
 
     cd singularity
 
     cd singularity
 
     ./autogen.sh
 
     ./autogen.sh
     ./configure --prefix=/usr/local
+
     ./configure --prefix= # System root to avoid changing sudo secure_path
 
     make
 
     make
 
     sudo make install
 
     sudo make install
Line 100: Line 101:
 
vagrant ssh
 
vagrant ssh
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
= Creating your singularity image =
 +
 +
When you log in with <code>vagrant ssh</code> above, your shell prompt should look similar to:
 +
 +
[vagrant@localhost ~]$
 +
 +
We are inside our CentOS virtual machine.
 +
 +
To create the raw image file use the <code>create</code> command.
 +
 +
<syntaxhighlight lang="bash">
 +
sudo singularity create --size 2048 centos7-container.img
 +
</syntaxhighlight>
 +
 +
We now need to install our software inside the image file by writing a "definition" file.
 +
You might be able to use one of several community contributed definition files are in the main singularity repository:
 +
[https://github.com/singularityware/singularity/tree/master/examples/contrib singularity/examples/contrib]
 +
The singularity [http://singularity.lbl.gov/docs-quick-start-installation user documentation] also has more detail,
 +
but for now we will look at a small example here.
 +
 +
As an example here, we will install R with the rstan package.
 +
Create the following <code>centos7-container.def</code> file using your favorite command-line text editor:
 +
 +
<syntaxhighlight lang="bash" line>
 +
# -*- mode: rpm-spec -*-
 +
 +
# Contents of file "centos7-container.def"
 +
 +
BootStrap: yum
 +
OSVersion: 7
 +
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
 +
Include: yum
 +
 +
# The default command to run once our image is finished.
 +
%runscript
 +
    Rscript $@
 +
   
 +
# Installing the software in our image.
 +
%post
 +
    yum -y install epel-release
 +
    yum -y install R
 +
    #Rscript -e 'require("rstan") | install.packages("rstan", repo="http://cran.rstudio.com/")'
 +
    #echo "Installed R with rstan package"
 +
 +
# Command
 +
%test
 +
    R --version
 +
</syntaxhighlight>
 +
 +
Now we run the <code>%post</code> software installation section with the singularity <code>bootstrap</code> command:
 +
<syntaxhighlight lang="bash">
 +
sudo singularity bootstrap centos7-container.{img,def}
 +
</syntaxhighlight>
 +
 +
The first time singularity runs the bootstrap process,
 +
you will see it installing some extra packages before it does anything in your <code>%post</code> steps.
 +
It is setting up a minimal operating system inside the image.
  
 
[[Category:Software]]
 
[[Category:Software]]

Revision as of 14:10, 12 May 2017

Singularity
Author Gregory M. Kurtzer and others
Website singularity.lbl.gov
Source GitHub
Category Container, Commandline utility
Help documentation
mailing list
PLoS paper



Singularity allows complex software to be run on the cluster, that would otherwise be difficult or impossible to install. Using singularity, you can generate a single "image" file in which you install all your software, and then you can copy that single image file to the cluster and run it there. Additionally, singularity allows more directly comparisons of running software between different clusters, because you can use the exact same versions of software that may not be installed on the different clusters.

This guide will help you create your own singularity images.

Requirements:

  • Your computer with macOS, Windows, or a GNU/Linux operating system.
  • VirtualBox to create a virtual machine in which we will install singularity.
  • Vagrant (version 1.9 or later) to greatly simplify administration of our virtual machine.
  • At least 40 GB of disk space.

Getting started

To be able to create and edit a singularity image, you must have root administrator access on a machine. Therefore you cannot manage singularity containers directly on the cluster; you must install your own copy of singularity on your computer. Singularity uses the Linux kernel to run the container, therefore you need to use a GNU/Linux machine for which we will use a virtual machine. For best results, it also helps to use a Linux distribution similar to the environment in which we will finally run our singularity image. As our cluster uses RHEL 6.7, we will create our singularity image using CentOS which, for our purposes, is functionally identical to RHEL.

If you are using macOS, you can install singularity via homebrew as explained on the singularity page [1] but which we will repeat here for completeness:

# Only run these commands if you are using macOS!

# Install Brew if you do not have it installed already
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

# The next commands will install Vagrant and the necessary bits
brew cask install virtualbox
brew cask install vagrant
brew cask install vagrant-manager

If you are using GNU/Linux, you can install virtualbox and vagrant through your package manager. You may need to log out and log in to your system again for your user account to be recognized as being in the vboxusers group.

Windows users can install virtualbox and vagrant directly from the main websites.

Now that we have the necessary tools, we can create our CentOS virtual machine in which we will install singularity:

# Create a working directory for the Vagrant configuration and
# generate a template Vagrantfile for "centos/7"
mkdir vm-singularity
cd vm-singularity
vagrant init centos/7
vagrant box add --provider virtualbox centos/7  # If this fails, you might need to upgrade vagrant

# Build and start the Vagrant hosted VM
vagrant up --provider virtualbox

# Run the necessary commands within the VM to install Singularity
vagrant ssh -c /bin/sh <<EOF
    sudo yum update
    sudo yum -y install @'development tools' emacs-nox
    git clone https://github.com/singularityware/singularity.git
    cd singularity
    ./autogen.sh
    ./configure --prefix=  # System root to avoid changing sudo secure_path
    make
    sudo make install
EOF

# Singularity is installed in your Vagrant CentOS VM! Now you can
# use Singularity as you would normally by logging into the VM
# directly
vagrant ssh

Creating your singularity image

When you log in with vagrant ssh above, your shell prompt should look similar to:

[vagrant@localhost ~]$

We are inside our CentOS virtual machine.

To create the raw image file use the create command.

sudo singularity create --size 2048 centos7-container.img

We now need to install our software inside the image file by writing a "definition" file. You might be able to use one of several community contributed definition files are in the main singularity repository: singularity/examples/contrib The singularity user documentation also has more detail, but for now we will look at a small example here.

As an example here, we will install R with the rstan package. Create the following centos7-container.def file using your favorite command-line text editor:

 1 # -*- mode: rpm-spec -*-
 2 
 3 # Contents of file "centos7-container.def"
 4 
 5 BootStrap: yum
 6 OSVersion: 7
 7 MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
 8 Include: yum
 9 
10 # The default command to run once our image is finished.
11 %runscript
12     Rscript $@
13     
14 # Installing the software in our image.
15 %post
16     yum -y install epel-release
17     yum -y install R
18     #Rscript -e 'require("rstan") | install.packages("rstan", repo="http://cran.rstudio.com/")'
19     #echo "Installed R with rstan package"
20 
21 # Command
22 %test
23     R --version

Now we run the %post software installation section with the singularity bootstrap command:

sudo singularity bootstrap centos7-container.{img,def}

The first time singularity runs the bootstrap process, you will see it installing some extra packages before it does anything in your %post steps. It is setting up a minimal operating system inside the image.