Difference between revisions of "Archiving from login nodes"

From Storrs HPC Wiki
Jump to: navigation, search
 
Line 1: Line 1:
 
These are instructions for archiving data from the HPC Cluster disks to writable archive.
 
These are instructions for archiving data from the HPC Cluster disks to writable archive.
 
    
 
    
(Note that normally we would recommend using Globus for this, but currently our local globus server (the DTN) will not write to our archive, so we are promoting this work-around)
+
'''Note''': normally we would recommend using Globus, but when Globus is not functioning correctly, you can try this work-around.
  
 
# Login to one of the login nodes<pre>ssh YOUR-NETID@login.storrs.hpc.uconn.edu</pre>
 
# Login to one of the login nodes<pre>ssh YOUR-NETID@login.storrs.hpc.uconn.edu</pre>
 
# Make a note of the hostname of the login node - either cn01, cn02, or cn03.  You'll need this information later.
 
# Make a note of the hostname of the login node - either cn01, cn02, or cn03.  You'll need this information later.
 
# Open a screen session with the following command (the option ''-S transfer'' simply assigns the name ''transfer'' to the screen session to make it easy to identify later): <pre> screen -S transfer </pre>
 
# Open a screen session with the following command (the option ''-S transfer'' simply assigns the name ''transfer'' to the screen session to make it easy to identify later): <pre> screen -S transfer </pre>
# Use ''rsync'' to copy your files.  I recommend you use the -a or -t option so that if your ''rsync'' session is somehow interrupted, it can be resumed afterwards: <pre> rsync -rt /scratch/YOUR-SOURCE-DIR  /archive/YOUR-DESTINATION-DIR/  </pre> '''NOTE''':  Do not modify the above command by appending a slash to YOUR-SOURCE-DIR above.  It will change the meaning of the command - this is an idiosyncrasy of the ''rsync'' command.<br>'''NOTE''': you can add the -v option to ''rsync'' to have it show each file as it transfers.
+
# Use ''rsync'' to copy your files.  We recommend you use the -a or -t option so that if your ''rsync'' session is somehow interrupted, it can be resumed afterwards: <pre> rsync -rt /scratch/YOUR-SOURCE-DIR  /archive/YOUR-DESTINATION-DIR/  </pre> '''NOTE''':  Do not modify the above command by appending a slash to YOUR-SOURCE-DIR above.  It will change the meaning of the command - this is an idiosyncrasy of the ''rsync'' command.<br>'''NOTE''': you can add the -v option to ''rsync'' to have it show each file as it transfers.
 
# To exit the screen session, hit [Ctrl-A] [Ctrl-D].  The ''rsync'' command will continue to run.  You can now log out of the login node if you choose.
 
# To exit the screen session, hit [Ctrl-A] [Ctrl-D].  The ''rsync'' command will continue to run.  You can now log out of the login node if you choose.
# To reconnect to the screen session, first, log into the same login node as where you ran your screen session.<pre>ssh YOUR-NETID@loginN.storrs.hpc.uconn.edu</pre>  The value for ''N'' in ''loginN'' in this ''ssh'' command will be either 1, 2 or 3, depending on the hostname you saw in step 2 above.  Then, once logged in, either type ''screen -x'', or if you have more than one screen session running (not likely unless you starting using screen a lot), run ''screen -ls'' to list the current screen sessions and then run ''screen -r NNNNN'' to connect to that session.  You can use [Ctrl-A] [Ctrl-D] again to disconnect from the session if you need to keep it running.
+
# To reconnect to the screen session, first, log into ''the same login node'' as where you ran your screen session.<pre>ssh YOUR-NETID@loginN.storrs.hpc.uconn.edu</pre>  The value for ''N'' in ''loginN'' in this ''ssh'' command will be either 1, 2 or 3, depending on the hostname you saw in step 2 above.  Then, once logged in, either type ''screen -x'', or if you have more than one screen session running (not likely unless you are using ''screen'' a lot), run ''screen -ls'' to list the current screen sessions and then run ''screen -r NNNNN'' to connect to that session.  You can use [Ctrl-A] [Ctrl-D] again to disconnect from the session if you need to keep it running.
# Once you are done with the screen session, you can simply type ''exit'' from inside the session to close it.
+
# Once you are done with the screen session, you can simply type ''exit'' from inside the ''screen'' session to close it.

Latest revision as of 16:20, 28 June 2021

These are instructions for archiving data from the HPC Cluster disks to writable archive.

Note: normally we would recommend using Globus, but when Globus is not functioning correctly, you can try this work-around.

  1. Login to one of the login nodes
    ssh YOUR-NETID@login.storrs.hpc.uconn.edu
  2. Make a note of the hostname of the login node - either cn01, cn02, or cn03. You'll need this information later.
  3. Open a screen session with the following command (the option -S transfer simply assigns the name transfer to the screen session to make it easy to identify later):
     screen -S transfer 
  4. Use rsync to copy your files. We recommend you use the -a or -t option so that if your rsync session is somehow interrupted, it can be resumed afterwards:
     rsync -rt /scratch/YOUR-SOURCE-DIR  /archive/YOUR-DESTINATION-DIR/  
    NOTE: Do not modify the above command by appending a slash to YOUR-SOURCE-DIR above. It will change the meaning of the command - this is an idiosyncrasy of the rsync command.
    NOTE: you can add the -v option to rsync to have it show each file as it transfers.
  5. To exit the screen session, hit [Ctrl-A] [Ctrl-D]. The rsync command will continue to run. You can now log out of the login node if you choose.
  6. To reconnect to the screen session, first, log into the same login node as where you ran your screen session.
    ssh YOUR-NETID@loginN.storrs.hpc.uconn.edu
    The value for N in loginN in this ssh command will be either 1, 2 or 3, depending on the hostname you saw in step 2 above. Then, once logged in, either type screen -x, or if you have more than one screen session running (not likely unless you are using screen a lot), run screen -ls to list the current screen sessions and then run screen -r NNNNN to connect to that session. You can use [Ctrl-A] [Ctrl-D] again to disconnect from the session if you need to keep it running.
  7. Once you are done with the screen session, you can simply type exit from inside the screen session to close it.