Difference between revisions of "GPU Guide"

From Storrs HPC Wiki
Jump to: navigation, search
(Also add to software category for visibility.)
m (Changed names to reflect new hostname)
Line 82: Line 82:
 
   /apps2/cuda/7.0.28/samples/1_Utilities/deviceQuery/deviceQuery # replaced {COMMAND}, you can put any commands you want here.
 
   /apps2/cuda/7.0.28/samples/1_Utilities/deviceQuery/deviceQuery # replaced {COMMAND}, you can put any commands you want here.
 
  $ sbatch gpu.sh
 
  $ sbatch gpu.sh
===Using nsight in the Hornet (not recommend)===
+
===Using nsight on the the cluster(not recommend)===
 
To use the eclipse-like tool, nsight supported by CUDA, you need the [[X|X Window]] to be available on your computer. When you login with the X Window feature, please follow:
 
To use the eclipse-like tool, nsight supported by CUDA, you need the [[X|X Window]] to be available on your computer. When you login with the X Window feature, please follow:
 
  $ module load cuda/7.0
 
  $ module load cuda/7.0

Revision as of 17:06, 19 February 2016

The following assumes you are already connected to the cluster.

Basic CUDA compilation

For some types of GPU work like compiling code, one needs to load the CUDA module.

One can list the available CUDA versions using:

module available cuda

At the time of writing version 7.0 is the latest, so we can load it using:

module load cuda/7.0

If you plan on doing extensive GPU work, you'll want to initadd it, which will load it on every log-in:

module initadd cuda/7.0

To compile with CUDA using the NVidia CUDA compiler:

nvcc {MYFILE.cu} -o {OUTPUT_FILE}

List GPU capabilities

Assign one of the GPU nodes using fisbatch:

module load cuda/7.0
fisbatch -p Westmere -c <numprocs> --gres=gpu:<numgpus>

where <numprocs> needs to be replaced by the number of processors you need, and <numgpus> needs to be replaced by the number of gpus you need.

Reminder: Each GPU node has 12 processors and 8 GPU cores.

Then run the deviceQuery sample program to get useful information about the GPU information like memory, processor cores, etc:

/apps2/cuda/7.0.28/samples/1_Utilities/deviceQuery/deviceQuery

For example:

module load cuda/7.0
fisbatch -p Westmere -c 1 --gres=gpu:1
FISBATCH -- the maximum time for the interactive screen is limited to 6 hours. You can add QoS to overwrite it.
FISBATCH -- waiting for JOBID 20248 to start on cluster=cluster and partition=Westmere
!
FISBATCH -- Connecting to head node (cn35)
[xxx00000@cn35 ~]$ /apps2/cuda/7.0.28/samples/1_Utilities/deviceQuery/deviceQuery
/apps2/cuda/7.0.28/samples/1_Utilities/deviceQuery/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Tesla M2050"
  CUDA Driver Version / Runtime Version          7.0 / 7.0
  CUDA Capability Major/Minor version number:    2.0
  ...
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.0, CUDA Runtime Version = 7.0, NumDevs = 1, Device0 = Tesla M2050
Result = PASS
[xxx00000@cn35 ~]$ exit
[screen is terminating]
Connection to cn35 closed.
FISBATCH -- exiting job

NOTE: please DO NOT FORGET to EXIT from the nodes so that the other users can use it.

Running GPU jobs

  • If you need to use just 1 GPU in EXCLUSIVE mode, your script (e.g. gpu.sh) should be:
 #!/bin/bash
 #SBATCH -o <stdout> 
 #SBATCH -e <stderr>
 #SBATCH --gres=gpu:1
 
 {COMMAND}

The submit the script by sbatch:

sbatch gpu.sh
  • Else, if you need between 2 or more GPU's in EXCLUSIVE mode, your script (e.g. gpu.sh) should be:
 #!/bin/bash
 #SBATCH -o <stdout> 
 #SBATCH -e <stderr>
 #SBATCH --gres=gpu:<numgpus>
 
 {COMMAND}

The submit the script by sbatch:

sbatch gpu.sh

Here all the stdout results of the {COMMAND} will go to the file <stdout> and all the stderr results will go to the file <stderr>.

For example, this will use 8 available GPU's in the cluster in EXCLUSIVE mode:

$ cat gpu.sh
 #!/bin/bash
 #SBATCH -o gpu-%J.out # replaced <stdout>
 #SBATCH -e gpu-%J.out # replaced <stderr>, the filename can ben different from <stdout>
 #SBATCH --gres=gpu:8 # replaced <numgpus>
 
 /apps2/cuda/7.0.28/samples/1_Utilities/deviceQuery/deviceQuery # replaced {COMMAND}, you can put any commands you want here.
$ sbatch gpu.sh

Using nsight on the the cluster(not recommend)

To use the eclipse-like tool, nsight supported by CUDA, you need the X Window to be available on your computer. When you login with the X Window feature, please follow:

$ module load cuda/7.0
$ fisbatch -c <numporcs> -p Westmere --gres=gpu:<numgpus>
FISBATCH -- the maximum time for the interactive screen is limited to 6 hours. You can add QoS to overwrite it.
FISBATCH -- waiting for JOBID 20248 to start on cluster=cluster and partition=Westmere
!
FISBATCH -- Connecting to head node (cn35)
[xxx00000@cn35 ~]$ nsight

Then, the nsight GUI will pop up. When you finish using nsight, please DO NOT FORGET to EXIT from the nodes so that the other users can use it.

[xxx00000@cn35 ~]$ exit
[screen is terminating]
Connection to cn35 closed.
FISBATCH -- exiting job

Other resources