Globus Connect

From Storrs HPC Wiki
Revision as of 11:16, 18 January 2017 by Drc12003 (talk | contribs) (Created page with "==GLOBUS user guide for STORRS-HPC== From [https://docs.globus.org/how-to/ Globus How-To] ===Key Concept: Globus Endpoint=== A Globus endpoint is a data transfer location....")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

GLOBUS user guide for STORRS-HPC

From Globus How-To

Key Concept: Globus Endpoint

A Globus endpoint is a data transfer location. Different types of resources (campus storage systems, HPC clusters, laptops, Amazon S3 buckets, scientific instruments, etc.) can be set up as Globus endpoints and made accessible to authorized users via Globus. An endpoint allows users with accounts on the underlying resource to transfer files, and it can also be configured to allow data sharing, publication, and discovery capabilities for other Globus users, including those who do not have local accounts on that resource.

There are two primary classes of endpoints: server endpoints and personal endpoints. Multi-user resources such as campus storage systems and HPC clusters are set up as server endpoints, typically by system administrators. Laptops or other personal computers are set up as personal endpoints, typically by individuals. Globus Connect is used to set up server and personal endpoints.

Create a globus personal account.

Complete the form at globusid.org/create and click "Create ID". (Your Globus ID should be based your NetID, if possible. Your email can be your @uconn.edu email or other.)

Gs-signup-2.png

You will receive a verification email. Copy the verification code from this email. Complete the account creation process by pasting the verification code into the field on the Verify E-mail Address page and click "Verify".

Gs-signup-3.png

Gs-signup-4.png

Once you’ve logged in, Globus will ask if you’d like to link to an existing account. If this is your first time signing up to use Globus (you don’t already have an existing account), you’ll click "No thanks, continue".

Gs-signup-5.png

Then you will need to allow Globus to use your Globus ID to view information and perform actions on your behalf within the service.

Gs-signup-6.png

Create a Globus Connect Personal Endpoint

Download and install Globus Connect Personal Endpoint on your workstation.

Log onto www.globus.org using your newly created Globus ID (if logged out) and go to "Manage Data." Click "Endpoints administered by me." Click "Add Globus Connect Personal Endpoint" and follow the online directions. Or, go to one of the following links and follow the directions:

Mac: Globus Connect Personal - Mac

Windows: Globus Connect Personal - Windows

Linux: Globus Connect Personal - Linux

Transfer Files

The Transfer Files page

After you’ve signed up, you’ll be sent to the Transfer Files page, or you can select Transfer Files from the Manage Data submenu.

Gs-transfer-1.png

The first time you navigate to the Transfer Files page, all fields will be blank.

Gs-transfer-2.png

Request a data transfer

When you click in either "Endpoint" field, you will receive a popup. To select your newly created personal endpoint, click the "Administered by me" tab and select your personal endpoint. To select a remote endpoint, you may search for it by typing the name in the "Endpoint" search field. NOTE: to select the UConn data transfer node, search for "uconnhpc#dtn-transfer." Login to uconn#dtn-transfer using your NetID.

Use the Transfer Files page to select the source and destination endpoints and paths, choose files to transfer, specify the transfer settings, and request the transfer with one of the arrows.

The Transfer Files screenshot shown here shows generic selections for your first data transfer. Selection descriptions follow the screenshot.

Gs-transfer-3.png

In the left panel, you can access files on the source endpoint. You may specify the path to your files; for example, if you are transferring from your Storrs HPC home directory, specify /gpfs/gpfs2/home/<your_username>. Files you select for transfer will be highlighted.

In the right panel, specify your destination path on the destination endpoint.

Note that you must hit "Enter" or click on the "Go" button after typing an endpoint path—simply moving the cursor to the next field is not sufficient.

Near the bottom, there are also various transfer settings that Globus supports listed. By default, Globus verifies file integrity after transfer using checksums. Click on the question mark icons for explanations of the other transfer settings. Globus gives you powerful control over the behavior of the transfer with a simple mouse click—change the transfer settings if you’d like. You may also enter a label for the transfer, but this is not required.

When you are ready to request the transfer, select a directional arrow between the endpoints. In this example, data will be transferred from the endpoint shown in the left pane to the endpoint shown in the right pane, so the right arrow is selected. A green notification panel will appear, confirming that the transfer request has been submitted.

Gs-transfer-4.png

Tip: Key Concept: Fire And Forget Data Transfer

After you request a file transfer, Globus takes over and does the work on your behalf. You can navigate away from the Transfer Files page, close the browser window, and even logout. Globus will optimize the transfer for performance, monitor the transfer for completion and correctness, and recover from network errors and endpoint resource downtime.

The Globus service routinely achieves 99.9% availability, providing nearly uninterrupted oversight of data transfers taking place on much less reliable networks and endpoint resources. When a problem is encountered part-way through the transfer, Globus resumes from the point of failure and does not retransmit all of the data specified in the original request.

Globus can handle extremely large data transfers, even those that do not complete within the authentication expiration period of an endpoint (which is controlled by the resource administrator). If the credentials expire before the transfer completes, Globus will notify you to re-authenticate on the endpoint so that Globus can continue the transfer. These wide-ranging capabilities make data transfer with Globus truly "fire and forget".

Confirm transfer completion

In our simple example only two small files were transferred, so the transfer will complete quickly. When it does, you will see an Activity notice at the top of your Transfer Files page. You can click the in the recent activity box to go to the Activity page. On the Activity page, click on the three dot icon on the right to view details about the transfer. You will also receive an email with the transfer details.

Gs-transfer-5.png

Gs-transfer-6.png

You may notice that the transferred files are not listed in the right pane of your Transfer Files page under the Globus Tutorial Endpoint 2 endpoint and the /~/ path, even though the transfer has completed. Refresh the list to see the updated contents.

NOTE: You may see connection failed messages in the Activity pane. This is not necessariyl a failure; check the destination folder on your destination endpoint outside of Globus and verify the data was indeed transferred. The error may be a callback function failure due to the client's firewall settings.

Gs-transfer-7.png