Globus Connect

From Storrs HPC Wiki
Jump to: navigation, search

Introduction

Globus is an easy-to-use, high-performance data transfer tool developed by the Computation Institute, which is a partnership between The University of Chicago and Argonne National Laboratory. UConn has deployed Globus as one mechanism to facilitate data transfer to and from the Storrs HPC cluster. You can also use Globus to transfer data between external institutions and UConn, including to and from your personal machine. Data can be transferred using a convenient web interface or a command line interface.

After you initiate a data transfer through the Globus interface, UConn's Globus node takes over and does the work on your behalf. You can navigate away from the Globus web page, close the browser window, and even logout. Globus will optimize the transfer for performance, monitor the transfer for completion and correctness, and recover from network errors and endpoint resource downtime. The Globus service is highly reliable, providing nearly uninterrupted oversight of data transfers. If a problem is encountered part-way through a transfer, Globus resumes from the point of failure and does not retransmit all of the data specified in the original request. Globus can handle extremely large data transfers, even those that do not complete within the authentication expiration period of an endpoint. If your credentials expire before a transfer completes, Globus will notify you to re-authenticate on the endpoint so that it can continue the transfer. These wide-ranging capabilities make data transfer with Globus truly "fire and forget".

Get Started

Globus Web App is accessible through any web browser. In order to access Globus Web App, you must first create an account through the Globus Auth service. Globus Auth is integrated with UConn NetID, so the process is quick and convenient.

1. To get started, visit Globus Web App at this link: https://www.globus.org/app/transfer. In the search box under Use your existing organizational login, search for and select University of Connecticut, as shown in Figure 1. Then, click Continue.

Figure 1

2. Next, you will be redirected to UConn's NetID login web page, as shown in Figure 2. Enter your NetID and password, and then click Login.

Figure 2

3. Next, you will be asked to link your new account to an existing Globus ID. If you've used the Globus service before, or any related services such as XSEDE, you may have an existing Globus ID. If so, click Link to an existing account, otherwise click No thanks, continue, as shown in Figure 3.

Figure 3

4. Next, you will be asked to confirm your personal details, and to optionally sign up for email updates from Globus, as shown in Figure 4. Click Continue.

Figure 4

5. Finally, you will be asked for permission to allow the Globus Web App to access the account you just created through the Globus Auth service, as shown in Figure 5. Click Allow.

Figure 5

Now that you have a working account on Globus, follow one of the scenarios below to initiate a data transfer.

Data Transfer Scenarios

In addition to the scenarios described below, you may find it helpful to read the official Globus Getting Started guide here: https://docs.globus.org/how-to/get-started/.

Transfer Data Between HPC File Systems

The instructions below demonstrate how to move data from the /shared file system to the /archive file system. The same instructions could be used to transfer data between any two file systems which are permanently mounted on the cluster. UConn's Globus node has access to the following file systems:

Path Description
/home/<user_name> User home directories
/shared/<group_name> Group shares
/archive/<share_name> Long term data storage
/scratch High-performance for temporary files

1. Login to the Globus Web App using your NetID and password. The left pane of the interface is the source side, from which files will be transferred. The right pane of the interface is the destination side, to which files will be transferred.

2. On the left pane, click inside the Endpoint box. Search for and select UConn Storrs HPC, which is the identifier for UConn's Globus server.

3. If you haven't access Globus in a while, you will be asked to login with a UConn username/password. Under Please authenticate to access this endpoint enter your NetID next to Username, and then your NetID password, as shown in Figure 6.

Figure 6

4. In the left pane, next to Path, enter the absolute path of the folder containing the files you wish to transfer to another location. For example, /shared/mylab.

5. On the right pane, click inside the Endpoint box. Search for and select UConn Storrs HPC.

6. In the right pane, next to Path, enter the absolute path of the folder to which you would like to transfer the data selected in the left pane. For example, /archive/mylab.

7. In the bottom pane, next to Label This Transfer, enter a descriptive name for your transfer. This is used to monitor its progress later on.

8. In the left pane, select the files to transfer and click the right-hand arrow to move the data from the location on the left, to the location on the right, as shown in Figure 7.

Figure 7

9. You can click on the Activity tab at the top of the window to monitor the progress of your data migration. You will also receive an email alert when the migration is complete.

Transfer Data To or From Your Personal Computer

You can use Globus to transfer data to or from your personal computer, such as a workstation in your lab, a computer connected to instrumentation, or even your laptop when on the go. To do so, it is first necessary to install software on that computer called the Globus Connect Personal Endpoint. The Globus service provides quality documentation for installing and configuring this software in common scenarios:

After you've installed the Globus Connect Personal Endpoint, your computer will be available in the list of Endpoints on the Globus web interface. Follow the instructions in the first scenario to initiate a data transfer. When you click in either Endpoint field, you will receive a popup. To select your newly created personal endpoint, click the Administered by me tab and select your personal endpoint.

Command Line Interface

Globus also has a command line interface that you can use with SSH keys which allows transfers to be scripted. Documentation can be viewed at the following links:

Getting Started with the Command Line Interface (CLI)

Using the Command Line Interface (CLI)

CLI Beyond the Basics

Link Your NetID to a Globus ID

If you have an existing Globus ID, such as username@globusid.org, you can link your UConn NetID to it so that you can login conveniently with your university credentials. To do so, follow the steps below.

1. Login to the Globus Web App using your Globus ID. To do so, visit this page: https://www.globus.org/app/transfer. Click on the link Then use Globus ID to sign in, as shown in Figure 8.

Figure 8

2. Enter your Globus ID username and password and click Log in, as shown in Figure 9.

Figure 9

3. Click on the Account link at the top right of the Globus Web App, as highlighted in Figure 10.

Figure 10

4. Click Add Linked Identity, as shown in Figure 11.

Figure 11

5. In the search box, enter University of Connecticut and click Continue, as shown in Figure 12.

Figure 12

6. Finally, you will be redirected to UConn's NetID login web page, as shown in Figure 13. Enter your NetID and password, and then click Login. Your UConn NetID will now be linked to your Globus ID.

Figure 13