Difference between revisions of "Archiving from login nodes"
From Storrs HPC Wiki
| Line 1: | Line 1: | ||
These are instructions for archiving data from the HPC Cluster disks to writable archive. | These are instructions for archiving data from the HPC Cluster disks to writable archive. | ||
| − | + | '''Note''': normally we would recommend using Globus, but when Globus is not functioning correctly, you can try this work-around. | |
# Login to one of the login nodes<pre>ssh YOUR-NETID@login.storrs.hpc.uconn.edu</pre> | # Login to one of the login nodes<pre>ssh YOUR-NETID@login.storrs.hpc.uconn.edu</pre> | ||
# Make a note of the hostname of the login node - either cn01, cn02, or cn03. You'll need this information later. | # Make a note of the hostname of the login node - either cn01, cn02, or cn03. You'll need this information later. | ||
# Open a screen session with the following command (the option ''-S transfer'' simply assigns the name ''transfer'' to the screen session to make it easy to identify later): <pre> screen -S transfer </pre> | # Open a screen session with the following command (the option ''-S transfer'' simply assigns the name ''transfer'' to the screen session to make it easy to identify later): <pre> screen -S transfer </pre> | ||
| − | # Use ''rsync'' to copy your files. | + | # Use ''rsync'' to copy your files. We recommend you use the -a or -t option so that if your ''rsync'' session is somehow interrupted, it can be resumed afterwards: <pre> rsync -rt /scratch/YOUR-SOURCE-DIR /archive/YOUR-DESTINATION-DIR/ </pre> '''NOTE''': Do not modify the above command by appending a slash to YOUR-SOURCE-DIR above. It will change the meaning of the command - this is an idiosyncrasy of the ''rsync'' command.<br>'''NOTE''': you can add the -v option to ''rsync'' to have it show each file as it transfers. |
# To exit the screen session, hit [Ctrl-A] [Ctrl-D]. The ''rsync'' command will continue to run. You can now log out of the login node if you choose. | # To exit the screen session, hit [Ctrl-A] [Ctrl-D]. The ''rsync'' command will continue to run. You can now log out of the login node if you choose. | ||
| − | # To reconnect to the screen session, first, log into the same login node as where you ran your screen session.<pre>ssh YOUR-NETID@loginN.storrs.hpc.uconn.edu</pre> The value for ''N'' in ''loginN'' in this ''ssh'' command will be either 1, 2 or 3, depending on the hostname you saw in step 2 above. Then, once logged in, either type ''screen -x'', or if you have more than one screen session running (not likely unless you | + | # To reconnect to the screen session, first, log into ''the same login node'' as where you ran your screen session.<pre>ssh YOUR-NETID@loginN.storrs.hpc.uconn.edu</pre> The value for ''N'' in ''loginN'' in this ''ssh'' command will be either 1, 2 or 3, depending on the hostname you saw in step 2 above. Then, once logged in, either type ''screen -x'', or if you have more than one screen session running (not likely unless you are using ''screen'' a lot), run ''screen -ls'' to list the current screen sessions and then run ''screen -r NNNNN'' to connect to that session. You can use [Ctrl-A] [Ctrl-D] again to disconnect from the session if you need to keep it running. |
| − | # Once you are done with the screen session, you can simply type ''exit'' from inside the session to close it. | + | # Once you are done with the screen session, you can simply type ''exit'' from inside the ''screen'' session to close it. |
Latest revision as of 16:20, 28 June 2021
These are instructions for archiving data from the HPC Cluster disks to writable archive.
Note: normally we would recommend using Globus, but when Globus is not functioning correctly, you can try this work-around.
- Login to one of the login nodes
ssh YOUR-NETID@login.storrs.hpc.uconn.edu
- Make a note of the hostname of the login node - either cn01, cn02, or cn03. You'll need this information later.
- Open a screen session with the following command (the option -S transfer simply assigns the name transfer to the screen session to make it easy to identify later):
screen -S transfer
- Use rsync to copy your files. We recommend you use the -a or -t option so that if your rsync session is somehow interrupted, it can be resumed afterwards:
rsync -rt /scratch/YOUR-SOURCE-DIR /archive/YOUR-DESTINATION-DIR/
NOTE: Do not modify the above command by appending a slash to YOUR-SOURCE-DIR above. It will change the meaning of the command - this is an idiosyncrasy of the rsync command.
NOTE: you can add the -v option to rsync to have it show each file as it transfers. - To exit the screen session, hit [Ctrl-A] [Ctrl-D]. The rsync command will continue to run. You can now log out of the login node if you choose.
- To reconnect to the screen session, first, log into the same login node as where you ran your screen session.
ssh YOUR-NETID@loginN.storrs.hpc.uconn.edu
The value for N in loginN in this ssh command will be either 1, 2 or 3, depending on the hostname you saw in step 2 above. Then, once logged in, either type screen -x, or if you have more than one screen session running (not likely unless you are using screen a lot), run screen -ls to list the current screen sessions and then run screen -r NNNNN to connect to that session. You can use [Ctrl-A] [Ctrl-D] again to disconnect from the session if you need to keep it running. - Once you are done with the screen session, you can simply type exit from inside the screen session to close it.
