Archiving from login nodes

From Storrs HPC Wiki
Revision as of 16:04, 18 June 2021 by Jar02014 (talk | contribs)
Jump to: navigation, search

These are instructions for archiving data from the HPC Cluster disks to writable archive.

(Note that normally we would recommend using Globus for this, but currently our local globus server (the DTN) will not write to our archive, so we are promoting this work-around)

  1. Login to one of the login nodes
  2. Make a note of the hostname of the login node - either cn01, cn02, or cn03. You'll need this information later.
  3. Open a screen session with the following command (the option -S transfer simply assigns the name transfer to the screen session to make it easy to identify later):
     screen -S transfer 
  4. Use rsync to copy your files. I recommend you use the -a or -t option so that if your rsync session is somehow interrupted, it can be resumed afterwards:
     rsync -rt /scratch/YOUR-SOURCE-DIR  /archive/YOUR-DESTINATION-DIR/  
    NOTE: Do not modify the above command by appending a slash to YOUR-SOURCE-DIR above. It will change the meaning of the command - this is an idiosyncrasy of the rsync command.
    NOTE: you can add the -v option to rsync to have it show each file as it transfers.
  5. To exit the screen session, hit [Ctrl-A] [Ctrl-D]. The rsync command will continue to run. You can now log out of the login node if you choose.
  6. To reconnect to the screen session, first, log into the same login node as where you ran your screen session.
    The value for N in loginN in this ssh command will be either 1, 2 or 3, depending on the hostname you saw in step 2 above. Then, once logged in, either type screen -x, or if you have more than one screen session running (not likely unless you starting using screen a lot), run screen -ls to list the current screen sessions and then run screen -r NNNNN to connect to that session. You can use [Ctrl-A] [Ctrl-D] again to disconnect from the session if you need to keep it running.
  7. Once you are done with the screen session, you can simply type exit from inside the session to close it.