Difference between revisions of "Globus Connect"

From Storrs HPC Wiki
Jump to: navigation, search
(Explain that one needs to go to globus.org for steps 3 and 4.)
Line 6: Line 6:
 
=Get Started=
 
=Get Started=
  
To get started using Globus, you must first create a Globus ID.  
+
The Globus service is a web application, accessible through your browser, called Globus Web App. In order to access Globus Web App, you must first create an account through the Globus Auth service. Globus Auth is integrated with UConn NetID, so the process is quick and convenient.
  
==Create a Globus ID==
+
1. To get started, visit Globus Web App at this link: https://www.globus.org/app/transfer. In the search box under '''Use your existing organizational login''', search for and select '''University of Connecticut''', as shown in ''Figure 1''. Then, click '''Continue'''.
  
'''1.''' Complete the form at [https://globusid.org/create globusid.org] and click "Create ID". (Your Globus ID should be based your NetID, if possible. Your email can be your @uconn.edu email or other.)
+
[[File:globusauth1.png|thumb|Figure 1|none]]
  
[[File:globus-new-id.png|682px]]
+
2. Next, you will be redirected to UConn's NetID login web page, as shown in ''Figure 2''. Enter your NetID and password, and then click '''Login'''.
  
'''2.''' You will receive a verification email. Copy the verification code from this email. Complete the account creation process by pasting the verification code into the field on the Verify E-mail Address page and click "Verify".
+
[[File:globusauth2.png|thumb|Figure 2|none]]
  
[[File:Gs-signup-3.png|640px]]
+
3. Next, you will be asked to link your new account to an existing Globus ID. If you've used the Globus service before, or any related services such as XSEDE, you may have an existing Globus ID. If so, click '''Link to an existing account''', otherwise click '''No thanks, continue''', as shown in ''Figure 3''.
  
[[File:Gs-signup-4.png|640px]]
+
[[File:globusauth3.png|thumb|Figure 3|none]]
  
==Globus==
+
4. Next, you will be asked to confirm your personal details, and to optionally sign up for email updates from Globus, as shown in ''Figure 4''. Click '''Continue'''.
  
'''3.''' Now you can login to the main [https://www.globus.org/login globus.org/login] website with your new Globus ID.
+
[[File:globusauth4.png|thumb|Figure 4|none]]
Once you’ve logged in, Globus will ask if you’d like to link to an existing account. If this is your first time signing up to use Globus (you don’t already have an existing account), you’ll click "No thanks, continue".
 
  
[[File:Gs-signup-5.png|640px]]
+
5. Finally, you will be asked for permission to allow the Globus Web App to access the account you just created through the Globus Auth service, as shown in ''Figure 5''. Click '''Allow'''.
  
'''4.''' Then you will need to allow Globus to use your Globus ID to view information and perform actions on your behalf within the service.
+
[[File:globusauth5.png|thumb|Figure 5|none]]
  
[[File:Gs-signup-6.png|640px]]
+
Now that you have a working account on Globus, follow one of the scenarios below to initiate a data transfer.  
 
 
Now that you have a working Globus ID, follow one of the scenarios below to initiate a data transfer.  
 
  
 
=Data Transfer Scenarios=
 
=Data Transfer Scenarios=
Line 53: Line 50:
 
|-
 
|-
 
|<code>/scratch</code>||High-performance for temporary files
 
|<code>/scratch</code>||High-performance for temporary files
|-
 
|<code>/scratch2</code>||Old scratch files (read-only)
 
|-
 
|<code>/scratch3</code>||Old scratch files (read-only)
 
 
|}
 
|}
  
'''1.''' Login to [https://www.globus.org/app/transfer the Globus web site] using your Globus ID. The left pane of the interface is the ''source'' side, from which files will be transferred. The right pane of the interface is the ''destination'' side, to which files will be transferred.
+
1. Login to [https://www.globus.org/app/transfer the Globus Web App] using your NetID and password. The left pane of the interface is the ''source'' side, from which files will be transferred. The right pane of the interface is the ''destination'' side, to which files will be transferred.
  
'''2.''' On the left pane, click inside the ''Endpoint'' box. Search for and select <code>uconnhpc#dtn-transfer</code>, which is the identifier for UConn's Globus node.
+
2. On the left pane, click inside the '''Endpoint''' box. Search for and select <code>uconnhpc#dtn-transfer</code>, which is the identifier for UConn's Globus server.
  
'''3.''' If you haven't access Globus in a while, you will be asked to login with a UConn username/password. Under ''Please authenticate to access this endpoint'' enter your NetID next to Username, and then your NetID password. For example:
+
3. If you haven't access Globus in a while, you will be asked to login with a UConn username/password. Under '''Please authenticate to access this endpoint''' enter your NetID next to Username, and then your NetID password, as shown in ''Figure 6''.
  
[[File:globus1.png|468px]]
+
[[File:globus1.png|thumb|Figure 6|none]]
  
'''4.''' In the left pane, next to ''Path'', enter the absolute path of the folder containing the files you wish to transfer to another location. For example, <code>/shared/mylab</code>.
+
4. In the left pane, next to '''Path''', enter the absolute path of the folder containing the files you wish to transfer to another location. For example, <code>/shared/mylab</code>.
  
'''5.''' On the right pane, click inside the ''Endpoint'' box. Search for and select <code>uconnhpc#dtn-transfer</code>.
+
5. On the right pane, click inside the '''Endpoint''' box. Search for and select <code>uconnhpc#dtn-transfer</code>.
  
'''6.''' In the right pane, next to ''Path'', enter the absolute path of the folder to which you would like to transfer the data selected in the left pane. For example, <code>/archive/mylab</code>.
+
6. In the right pane, next to '''Path''', enter the absolute path of the folder to which you would like to transfer the data selected in the left pane. For example, <code>/archive/mylab</code>.
  
'''7.''' In the bottom pane, next to Label This Transfer, enter a descriptive name for your transfer. This is used to monitor its progress later on.
+
7. In the bottom pane, next to '''Label This Transfer''', enter a descriptive name for your transfer. This is used to monitor its progress later on.
  
'''8.''' In the left pane, select the files to transfer and click the right-hand arrow to move the data from the location on the left, to the location on the right. See the screenshot below for an example.
+
8. In the left pane, select the files to transfer and click the right-hand arrow to move the data from the location on the left, to the location on the right, as shown in ''Figure 7''.
  
[[File:globus2.png|969px]]
+
[[File:globus2.png|thumb|Figure 7|none]]
  
'''9.''' You can click on the Activity tab at the top of the window to monitor the progress of your data migration. You will also receive an email alert when the migration is complete.
+
9. You can click on the '''Activity''' tab at the top of the window to monitor the progress of your data migration. You will also receive an email alert when the migration is complete.
  
 
==Transfer Data To or From Your Personal Computer==
 
==Transfer Data To or From Your Personal Computer==

Revision as of 09:35, 5 May 2017

Introduction

Globus is an easy-to-use, high-performance data transfer tool developed by the Computation Institute, which is a partnership between The University of Chicago and Argonne National Laboratory. UConn has deployed Globus as one mechanism to facilitate data transfer to and from the Storrs HPC cluster. You can also use Globus to transfer data between external institutions and UConn, including to and from your personal machine. Data can be transferred using a convenient web interface or a command line interface.

After you initiate a data transfer through the Globus interface, UConn's Globus node takes over and does the work on your behalf. You can navigate away from the Globus web page, close the browser window, and even logout. Globus will optimize the transfer for performance, monitor the transfer for completion and correctness, and recover from network errors and endpoint resource downtime. The Globus service is highly reliable, providing nearly uninterrupted oversight of data transfers. If a problem is encountered part-way through a transfer, Globus resumes from the point of failure and does not retransmit all of the data specified in the original request. Globus can handle extremely large data transfers, even those that do not complete within the authentication expiration period of an endpoint. If your credentials expire before a transfer completes, Globus will notify you to re-authenticate on the endpoint so that it can continue the transfer. These wide-ranging capabilities make data transfer with Globus truly "fire and forget".

Get Started

The Globus service is a web application, accessible through your browser, called Globus Web App. In order to access Globus Web App, you must first create an account through the Globus Auth service. Globus Auth is integrated with UConn NetID, so the process is quick and convenient.

1. To get started, visit Globus Web App at this link: https://www.globus.org/app/transfer. In the search box under Use your existing organizational login, search for and select University of Connecticut, as shown in Figure 1. Then, click Continue.

Figure 1

2. Next, you will be redirected to UConn's NetID login web page, as shown in Figure 2. Enter your NetID and password, and then click Login.

Figure 2

3. Next, you will be asked to link your new account to an existing Globus ID. If you've used the Globus service before, or any related services such as XSEDE, you may have an existing Globus ID. If so, click Link to an existing account, otherwise click No thanks, continue, as shown in Figure 3.

Figure 3

4. Next, you will be asked to confirm your personal details, and to optionally sign up for email updates from Globus, as shown in Figure 4. Click Continue.

Figure 4

5. Finally, you will be asked for permission to allow the Globus Web App to access the account you just created through the Globus Auth service, as shown in Figure 5. Click Allow.

Figure 5

Now that you have a working account on Globus, follow one of the scenarios below to initiate a data transfer.

Data Transfer Scenarios

In addition to the scenarios described below, you may find it helpful to read the official Globus Getting Started guide here: https://docs.globus.org/how-to/get-started/.

Transfer Data Between HPC File Systems

The instructions below demonstrate how to move data from the /shared file system to the /archive file system. The same instructions could be used to transfer data between any two file systems which are permanently mounted on the cluster. UConn's Globus node has access to the following file systems:

Path Description
/home/<user_name> User home directories
/shared/<group_name> Group shares
/archive/<share_name> Long term data storage
/scratch High-performance for temporary files

1. Login to the Globus Web App using your NetID and password. The left pane of the interface is the source side, from which files will be transferred. The right pane of the interface is the destination side, to which files will be transferred.

2. On the left pane, click inside the Endpoint box. Search for and select uconnhpc#dtn-transfer, which is the identifier for UConn's Globus server.

3. If you haven't access Globus in a while, you will be asked to login with a UConn username/password. Under Please authenticate to access this endpoint enter your NetID next to Username, and then your NetID password, as shown in Figure 6.

Figure 6

4. In the left pane, next to Path, enter the absolute path of the folder containing the files you wish to transfer to another location. For example, /shared/mylab.

5. On the right pane, click inside the Endpoint box. Search for and select uconnhpc#dtn-transfer.

6. In the right pane, next to Path, enter the absolute path of the folder to which you would like to transfer the data selected in the left pane. For example, /archive/mylab.

7. In the bottom pane, next to Label This Transfer, enter a descriptive name for your transfer. This is used to monitor its progress later on.

8. In the left pane, select the files to transfer and click the right-hand arrow to move the data from the location on the left, to the location on the right, as shown in Figure 7.

Figure 7

9. You can click on the Activity tab at the top of the window to monitor the progress of your data migration. You will also receive an email alert when the migration is complete.

Transfer Data To or From Your Personal Computer

You can use Globus to transfer data to or from your personal computer, such as a workstation in your lab, a computer connected to instrumentation, or even your laptop when on the go. To do so, it is first necessary to install software on that computer called the Globus Connect Personal Endpoint. The Globus service provides quality documentation for installing and configuring this software in common scenarios:

After you've installed the Globus Connect Personal Endpoint, your computer will be available in the list of Endpoints on the Globus web interface. Follow the instructions in the first scenario to initiate a data transfer. When you click in either Endpoint field, you will receive a popup. To select your newly created personal endpoint, click the Administered by me tab and select your personal endpoint.

Command Line Interface

Globus also has a command line interface that you can use with SSH keys which allows transfers to be scripted. Documentation can be viewed at the following links:

Getting Started with the Command Line Interface (CLI)

Using the Command Line Interface (CLI)

CLI Beyond the Basics