The ACF provides several ways for transferring files to/from the NFS home directories, NFS project directories, Lustre project directories, and Lustre scratch directories. DTNs (Data Transfer Nodes) furnish this capability. At the time of this writing, there are four DTNs available to ACF users. The table below shows these nodes.
|Data Transfer Node||IP Address||Authentication Supported||File Transfer Protocols Supported||File System Access|
|SCP, SFTP, Globus||Home,|
|SCP, SFTP, Globus||Home,|
|SCP, SFTP, Globus||Home,|
|SCP, SFTP, Globus||Home,|
The listed DTNs are setup for NetID authentication, Duo TFA, and authentication through an InCommon Credential so users can login to this node and perform data transfer functions. To connect to these DTNs, use ssh in a terminal. More information on ssh usage can be found in the Access and Login document. Replace the hostname of the login node with the hostname of the DTN to which you wish to connect, then authenticate with your UT NetID, password, and Duo TFA.
SCP and SFTP
SCP and SFTP are both ssh utilities available for transferring files on the ACF. However, they perform slower than Globus. At the time of this writing, Globus offers the fastest data transfers on the ACF. Still, SCP and SFTP are useful for quick, small transfers. For larger file transfers, please use Globus.
SCP and SFTP are available to Linux and MacOS systems by default. Windows 10 users with the most recent updates can use these utilities within Command Prompt or PowerShell. Windows 7 and 8 users must use a third-party utility to use SCP and SFTP. For more information on ssh in Windows, see the Access and Login document. For Windows 7 and 8 users, the third-party utilities FileZilla and WinSCP are reviewed later in this document.
The general syntax of SCP is given below. In general, SCP is useful when transferring a file on your system to the ACF. The <source> argument is the pathname of the file on your system that you wish to copy. The <destination> (in this case, datamover1) argument is the hostname of the datamover you wish to use. Additionally, the <directory> argument specifies the absolute pathname within the destination to place the file.
scp <source> <NetID>@datamover1.acf.utk.edu:<directory>
If you wanted to copy a file from your system and place it on the ACF, you could use
For SFTP, you specify the hostname of the system to which you intend to connect. For example, to securely transfer files between your local system and the ACF, use the syntax below in a terminal on your local system. Ensure that you enter SFTP from the directory that contains the file(s) you wish to copy to the ACF. You can use the pwd command to determine your current directory before entering SFTP.
Once you authenticate with your UT NetID, password, and Duo TFA, you will enter SFTP’s interactive mode. Use the
put <file> command to upload a file to the ACF. For example, to upload a file named JobScript.sh to the ACF from your local machine, use
put JobScript.sh. This syntax assumes that the JobScript.sh file is in the directory from which you entered SFTP.
To retrieve files from the ACF, use the
get <file> command. To download a file named ResearchResults.txt from the ACF to your local machine, use
get ResearchResults.txt. SFTP will place the file in the directory from which you entered the utility. To change directories on the ACF, use the
cd <directory> command. Use the
lcd <directory> command to change the directory on your local system. Once you are done with SFTP, use the
exit commands to exit it. Other commands are available with the SFTP utility. Type
help within SFTP to read more about them.
Globus Web-based Transfers
ACF users can access the Globus web interface to perform data transfers to and from ACF resources. To start, visit the Globus website and consult the Getting Started guide. Review the how-to documentation to understand the basics of Globus and its usage.
Another Globus feature is the Globus Connect Personal client software. Globus Connect Personal allows your system to act as an endpoint for Globus so that you can transfer files to and from your local machine to a remote one. This software can be obtained from the Globus website, though you will set up this software on your local machine as part of the Globus configuration process.
Please note that Globus only works with a proper UT CILogon InCommon credential. You must first set up this credential before you can use Globus. The steps below guide you through the process of configuring your credential for use on the ACF.
Configuring Your InCommon Credential
To use the Globus file transfer service, each user must perform the following actions:
- Associate their NetID with their NICS account.
- Associate their InCommon credential with their NICS account.
- Authenticate to the Globus web interface using the InCommon credential.
Both association options are highlighted in Figure 3.1. To start, login to the NICS User Portal. Upon successful login, click the UTK/UTHSC NetID association link. Follow the prompts. Next, click the InCommon credential association option. The process to associate your InCommon credential with your account is explained below.
To setup this credential, carefully follow these steps.
Step 1: Access the NICS User Portal, if you have not already.
Step 2: Select the option to associate your InCommon credential with your NICS account. Refer to Figure 3.1 to see these options.
Step 3: Select “Continue” when presented with the “About CILogon” page.
Step 4: On the CILogon page, select the University of Tennessee as the identity provider.
Step 5: Authenticate with your UT NetID, password, and Duo TFA.
Step 6: Enter a password for your new InCommon credential. Please record and store this password in a secure location for future reference.
Step 7: Download your credential. Click download, then save it to your local system.
Credential information is updated on the ACF DTNs every hour. If you are unable to use Globus, please wait an hour to allow the DTNs time to obtain your information. Generally, if you see your InCommon credential appear in the NICS user portal, you should be able to use Globus.
Using the Globus Web Interface
To access the Globus interface in your browser, navigate to the Globus website. Login using the existing organizational login option. Verify that the University of Tennessee is selected, then select “Continue.” Authenticate with your UT NetID, password, and Duo TFA. You will then see the interface depicted in Figure 3.6. If you experience issues logging in, verify that your InCommon credential was configured per the steps given in the Configuring your InCommon Credential section.
Before you can initiate file transfers between your local machine and the ACF, you must configure endpoints. One endpoint will reference your local system while the other will reference one of the ACF DTNs. Further instructions on these endpoints will be provided below.
To configure the endpoints in the Globus interface, select the “Endpoints” tab on the left-side of the page. You will then see a page similar to Figure 3.7. At the top-right of the page, select “Create new endpoint.” On the endpoint type selection page, choose “Globus Connect Personal.”
On the next page, name the endpoint. The name you choose is unimportant; however, it should be something memorable. After you name the endpoint, generate a setup key for the Globus Connect Personal client software. The option to generate the key is listed under Step 2 in Figure 3.8. Copy this key. Finally, download and install the Globus Connect Personal client software. When prompted, enter the setup key you copied to configure your local machine as an endpoint. Refer to Figure 3.8 for a screenshot of the endpoint creation page.
Once you configure your local machine as a Globus endpoint, return to the “File Manager” tab on the left-side of the page. Make sure you select the double panels option in the top-right of the page (Figure 3.9 highlights this option). This will display your local machine’s filesystem in addition to the datamover’s. Once both panels are displayed, click on “Collection” in the left panel. Type the name of your endpoint in the search bar or find it under “My Collections."
After your endpoint has been selected, you will return to the File Manager. In the right panel, click on “Collection.” Search for one of the four ACF datamovers. The hostnames of these DTNs are given below.
Once both endpoints are configured, you can transfer data between the two. You can select individual files and directories for these transfers. When you select the data you wish to transfer, press the “Start” button below the endpoint from which you will transfer data. Additionally, you can navigate throughout the filesystem hierarchy in either endpoint using the Globus interface. Other options are available for your transfers, but they are usually unnecessary for most transfers. Figure 3.9 shows what the Globus interface should look like when both endpoints are selected.
This X.509 distinguished name (DN) information is put into the /etc/grid-security/grid-mapfile on the ACF DTNs and this process is done every hour so you may have to wait an hour to use this authentication method. Once you have this setup and your credential is in the /etc/grid-security/grid-mapfile on the ACF DTNs you are ready to start using Globus for data transfers. If you want to use GSISCP, follow the instructions below. The ACF DTNs are configured to use CILogon OAuth credentials. For example, the nics#datamover1 Globus endpoint is setup to use your CILogon credential so just login to Globus, select the nics#datamover1 endpoint and authenticate with your CILogon password. No other authentication method will work for the ACF DTNs with Globus and the GSISCP protocols (one cannot use NetID and password, for example).
To use your new X.509 credential with GSISCP you will need to obtain a credential pem file and put it in your home directory on the ACF. The file specifically needs to go into the in ~/.globus/usercert.pem with permissions 600. If you didn't save the credential following the instructions above you can get a new credential pem file by going back to the https://cilogon.org/ page and go through the process again to generate a new certificate. This will then prompt you for a credential password so go ahead and type one in. Again, be sure to remember what this password is for future reference.
Once you have this credential in the ~/.globus/usercred.pem file then login to one of the ACF DTNs and run
grid-proxy-init. grid-proxy-init will prompt you for your CILogon credential password. This will create a proxy credential which can be used with GSISCP. Once you have done the grid-proxy-init you can then do a gsiscp without having to type a username or password. The default credential lifetime is 12 hours.
Setting up x.509 authentication
In order to use the GSISCP and Globus file transfer services each user needs to do three things:
- In the NICS portal associate their NetID with their NICS account (see the image below) and
- In the NICS portal setup their X.509 user certificate by associating their CILogon InCommon credential with their NICS account
- Authenticate to the Globus web-based interface for file transfers using the University of Tennessee X.509 based CILogon InCommon credential
To setup this credential you will select "University of Tennessee" as the identity provider and login using your University of Tennessee NetID username and password when prompted by the InCommon CILogon interface. You will set a password for your X.509 credential. Please note and remember this password as you will use it in setting up Globus or GSISCP with X.509 credentials. Once you go through the CILogon process the Distinguished Name (DN) of your X.509 credential will be associated with the NICS ACF infrastructure and will be available for use. Screeshots of the step by step process is shown below.
Step 0: Login to the Newton login node in order to save the credential you are about to create in Step 4
Step 1: select University of Tennessee as the Identity Provider
Step 2: Authenticate with your UT NetID and Password
Step 3: enter a password for your new InCommon credential (and remember this!)
Step 4: you will get a screen that shows you can click to download your certificate. Click to download and save locally. You could also use
wget to this URL from Newton to save to your Newton home directory. There is a time limit for access to this certificate so be aware of that. You may have to move quickly to download the certificate.
This X.509 distinguished name (DN) information is put into the /etc/grid-security/grid-mapfile on the SIP DTNs. Once you have this setup and your credential is in the /etc/grid-security/grid-mapfile on the DTNs you are ready to start using Globus for data transfers. If you want to use GSISCP you will need to follow the instructions in the below paragraph to set that up. The SIP DTNs are configured to use CILogon OAuth credentials. For the example, the nics#datamover1 Globus endpoint is setup to use your CILogon credential so just login to Globus, select the nics#datamover1 endpoint and authenticate with your CILogon password. No other authentication method will work for the SIP DTNs with Globus and the GSISCP protocols (one cannot use NetID and password, for example).
To use your new X.509 credential with GSISCP you will need to obtain a credential pem file and put it in your home directory. The file specifically needs to go into the in ~/.globus/usercert.pem with permissions 600. If you didn't save the credential following the instructions above you can get a new credential pem file by going back to the https://cilogon.org/ page and go through the process again to generate a new certificate. This will then prompt you for a credential password so go ahead and type one in. Again, be sure to remember what this password is for future reference. The CILogon page will give you a link to download the certificated needed as shown below.
Once you have this credential in the ~/.globus/usercred.pem file then login to one of the DTNs and run
grid-proxy-init will prompt you for your CILogon credential password. This will create a proxy credential which can be used with GSISCP. Once you have done the
grid-proxy-init you can then do a gsiscp without having to type a username or password. The default credential lifetime is 12 hours. See the following transcript for an example.
Using FileZilla to Transfer Files
FileZilla will work with file transfers to the ACF. Please only use the DTNs listed in Table 1.1 at the beginning of this document.
To use the FileZilla client with your NetID, password, and Duo TFA, follow the below steps.
Step 1: Open the FileZilla client.
Step 2: Select File, then Site Manager
Step 3: Select “New Site,” then provide the necessary information. For the host, select one of the datamovers listed in Table 1.1. For protocol, select SFTP - SSH File Transfer Protocol. For Logon Type, select Interactive. For User, type your UT NetID. Finally, rename the entry under sites from "New Site" to something more memorable, such as the name of the datamover you chose to use. Refer to Figures 4.2 and 4.3 to identify where to find these options.
Step 4: Select Transfer Settings, then check the box for Limit the number of simultaneous connections. Make sure the value beneath this checkbox is 1.
Step 5: Select “Connect” in the Site Manager window.
Step 6: When prompted, enter your password.
Step 7: When prompted, type a “1” to send a Duo Push to your mobile device, then authenticate with Duo TFA. Upon successful authentication, you will be logged in to the datamover through FileZilla.
Using WinSCP to Transfer Files
WinSCP can perform file transfers to and from the ACF. Please use the DTNs listed in Table 1.1 at the beginning of this document.
To use the WinSCP client with your NetID, password, and Duo TFA, follow these steps.
Step 1: Open WinSCP, then click on “New Site.”
Step 2: Provide the hostname of the datamover for “Host name,” your UT NetID for “User name,” and your password. Leave the port number as 22.
Step 3: When warned about an unknown server, select “Yes.”
Step 4: The authentication banner will appear. Select “Continue.”
Step 5: When prompted, type “1” to receive a Duo Push on your mobile device. Authenticate with Duo. You will then be logged in.
Step 6: Once you authenticate, you will get the WinSCP application screen. On the left side of the screen, you see your local machine. On the right side of the screen, you see the remote system into which you are logged.
Last Updated: 01 / 16 / 2020