Parallel Guide

From Storrs HPC Wiki
Jump to: navigation, search
GNU Parallel
Author Ole Tange
Source Savannah
Category Commandline utility
Help man page
mailing list

GNU Parallel is powerful, lightweight utility for running single-threaded (i.e. serial) programs multiple times with different inputs. It parallelizes software which which was not designed for parallel execution, and works with Slurm's srun. GNU Parallel achieves this by creating "slots" (set by the -j or --jobs option), and as jobs in slots are completed, it runs remaining jobs.

Some of the more powerful features are:

  • Tracking individual jobs in the --joblog
  • Pausing and resuming
  • Retrying failed jobs
  • Reading parameters from files or stdin
  • Parsing directory {//}, file {/.} and extension {.} names
  • Memory control with --memfree
  • Shebang line for scripts

To understand how to use some of these features, it's well worth spending an hour going through the tutorial.

Using the Parallel module with the scheduler

Clone our Git repository with the setup script and follow the instructions in the README to use GNU Parallel in your submission file: