Globus Connect

From Storrs HPC Wiki
Jump to: navigation, search

Introduction

Globus is an easy-to-use, high-performance data transfer tool developed by the Computation Institute, which is a partnership between The University of Chicago and Argonne National Laboratory. UConn has deployed Globus as one mechanism to facilitate data transfer to and from the Storrs HPC cluster. You can also use Globus to transfer data between external institutions and UConn, including to and from your personal machine. Data can be transferred using a convenient web interface or a command line interface.

After you initiate a data transfer through the Globus interface, UConn's Globus node takes over and does the work on your behalf. You can navigate away from the Globus web page, close the browser window, and even logout. Globus will optimize the transfer for performance, monitor the transfer for completion and correctness, and recover from network errors and endpoint resource downtime. The Globus service is highly reliable, providing nearly uninterrupted oversight of data transfers. If a problem is encountered part-way through a transfer, Globus resumes from the point of failure and does not retransmit all of the data specified in the original request. Globus can handle extremely large data transfers, even those that do not complete within the authentication expiration period of an endpoint. If your credentials expire before a transfer completes, Globus will notify you to re-authenticate on the endpoint so that it can continue the transfer. These wide-ranging capabilities make data transfer with Globus truly "fire and forget".

Get Started

To get started using Globus, you must first create a Globus ID.

Create a Globus ID

1. Complete the form at globusid.org and click "Create ID". (Your Globus ID should be based your NetID, if possible. Your email can be your @uconn.edu email or other.)

Globus-new-id.png

2. You will receive a verification email. Copy the verification code from this email. Complete the account creation process by pasting the verification code into the field on the Verify E-mail Address page and click "Verify".

Gs-signup-3.png

Gs-signup-4.png

Globus

3. Now you can login to the main globus.org/login website with your new Globus ID. Once you’ve logged in, Globus will ask if you’d like to link to an existing account. If this is your first time signing up to use Globus (you don’t already have an existing account), you’ll click "No thanks, continue".

Gs-signup-5.png

4. Then you will need to allow Globus to use your Globus ID to view information and perform actions on your behalf within the service.

Gs-signup-6.png

Now that you have a working Globus ID, follow one of the scenarios below to initiate a data transfer.

Data Transfer Scenarios

In addition to the scenarios described below, you may find it helpful to read the official Globus Getting Started guide here: https://docs.globus.org/how-to/get-started/.

Transfer Data Between HPC File Systems

The instructions below demonstrate how to move data from the /shared file system to the /archive file system. The same instructions could be used to transfer data between any two file systems which are permanently mounted on the cluster. UConn's Globus node has access to the following file systems:

Path Description
/home/<user_name> User home directories
/shared/<group_name> Group shares
/archive/<share_name> Long term data storage
/scratch High-performance for temporary files
/scratch2 Old scratch files (read-only)
/scratch3 Old scratch files (read-only)

1. Login to the Globus web site using your Globus ID. The left pane of the interface is the source side, from which files will be transferred. The right pane of the interface is the destination side, to which files will be transferred.

2. On the left pane, click inside the Endpoint box. Search for and select uconnhpc#dtn-transfer, which is the identifier for UConn's Globus node.

3. If you haven't access Globus in a while, you will be asked to login with a UConn username/password. Under Please authenticate to access this endpoint enter your NetID next to Username, and then your NetID password. For example:

Globus1.png

4. In the left pane, next to Path, enter the absolute path of the folder containing the files you wish to transfer to another location. For example, /shared/mylab.

5. On the right pane, click inside the Endpoint box. Search for and select uconnhpc#dtn-transfer.

6. In the right pane, next to Path, enter the absolute path of the folder to which you would like to transfer the data selected in the left pane. For example, /archive/mylab.

7. In the bottom pane, next to Label This Transfer, enter a descriptive name for your transfer. This is used to monitor its progress later on.

8. In the left pane, select the files to transfer and click the right-hand arrow to move the data from the location on the left, to the location on the right. See the screenshot below for an example.

Globus2.png

9. You can click on the Activity tab at the top of the window to monitor the progress of your data migration. You will also receive an email alert when the migration is complete.

Transfer Data To or From Your Personal Computer

You can use Globus to transfer data to or from your personal computer, such as a workstation in your lab, a computer connected to instrumentation, or even your laptop when on the go. To do so, it is first necessary to install software on that computer called the Globus Connect Personal Endpoint. The Globus service provides quality documentation for installing and configuring this software in common scenarios:

After you've installed the Globus Connect Personal Endpoint, your computer will be available in the list of Endpoints on the Globus web interface. Follow the instructions in the first scenario to initiate a data transfer. When you click in either Endpoint field, you will receive a popup. To select your newly created personal endpoint, click the Administered by me tab and select your personal endpoint.

Command Line Interface

Globus also has a command line interface that you can use with SSH keys which allows transfers to be scripted. Documentation can be viewed at the following links:

Getting Started with the Command Line Interface (CLI)

Using the Command Line Interface (CLI)

CLI Beyond the Basics