Lab Exercise: Data Management

Purpose:

During this lab the user will mange data on local and remote machines using several different tools provided by the toolkit.

1. Moving data and programs with SCP
2. Moving data and programs with globus-url-copy
3. Moving data and programs with UberFTP

Secure Copy (SCP)

1. Secure copy can take advantage of GSI enabled ssh, which means you will also be able to use scp without typing a password when transferring files!
   
   Before beginning, double check to make sure you have a valid proxy initialized (grid-proxy-info).  If not, initialize one with the grid-proxy-init command:

$ grid-proxy-info subject : /C=US/O=National Center for Supercomputing Applications/CN=Mike Freemon/CN=proxy issuer : /C=US/O=National Center for Supercomputing Applications/CN=Mike Freemon identity : /C=US/O=National Center for Supercomputing Applications/CN=Mike Freemon type : full legacy globus proxy strength : 512 bits path : /tmp/x509up_u500 timeleft : 7:59:51 $ grid-proxy-init Your identity: /C=US/O=National Center for Supercomputing Applications/CN=Mike Freemon Enter GRID pass phrase for this identity: Creating proxy ................................ Done Your proxy is valid until: Tue Feb 22 21:19:19 2005 $

2. First, create a file on the client machine called local_scp.txt and fill it with some text:

$ cd $ cat >local_scp.txt This is a test file for use with SCP put CNTL-D $ cat local_scp.txt This is a test file for use with SCP put $

3. Open a session on the server, and create a file call server_scp.txt:

$ ssh ldas-grid.ligo-la.caltech.edu [...snip...] $ cat >server_scp.txt This is a test file for use with SCP get CNTL-D $ cat server_scp.txt This is a test file for use with SCP get $ exit $

4. For copying  from  remote host to local machine, issue the following command (from the local machine):

$ scp ldas-grid.ligo-la.caltech.edu:server_scp.txt . server_scp.txt 100% 41 0.0KB/s 00:00 $ ls server_scp.txt $ cat server_scp.txt This is a test file for use with SCP get $

5. Now try copying the file you created on your local machine to the remote server.  Again, from the local machine, execute:

$ scp local_scp.txt ldas-grid.ligo-la.caltech.edu:local_scp.txt local_scp.txt 100% 41 0.0KB/s 00:00 $ ssh ldas-grid.ligo-la.caltech.edu [...snip...] $ ls local_scp.txt server_scp.txt $ cat local_scp.txt This is a test file for use with SCP put $ exit $

 

 

globus-url-copy

Information on globus-url-copy can be found at: http://www-unix.globus.org/toolkit/docs/3.2/gridftp/user/globusurlcopy.html

The basic syntax for globus-url-copy is: globus-url-copy [options]  <Source URL>  <Destination URL>. 

• Source URL: specifies the original URL.  If this is a directory, all the files in that directory will be copied.
• Destination URL: specifies the URL where the files will be copied.  If coping multiple files, this must be a directory.

6. On your local machine create a file called local_globus.txt and fill it with some text.

$ cd $ cat >local_globus.txt This is a test file from the local machine for globus-url-copy CNTL-D $ cat local_globus.txt This is a test file from the local machine for globus-url-copy $

7. Open a session to the server and create a file on it called server_globus.txt as follows:

$ ssh ldas-grid.ligo-la.caltech.edu [...snip...] $ cat >server_globus.txt This is a test file from the server for globus-url-copy CNTL-D $ cat server_globus.txt This is a test file from the server for globus-url-copy $ exit $

8. Finally, on the server, create a file called: party_globus.txt:

$ ssh ldas-grid.ligo-la.caltech.edu [...snip...] $ cat >party_globus.txt This is a test file for 3rd party transfer using globus-url-copy CNTL-D $ cat party_globus.txt This is a test file for 3rd party transfer using globus-url-copy $ exit $

9. To perform a copy from the server to the local machine, issue the command (while logged into the local machine):

$ globus-url-copy gsiftp://ldas-grid.ligo-la.caltech.edu/data2/ <userid>/server_globus.txt \                   file:/home/<userid>/server_globus.txt $ cat server_globus.txt This is a test file from the server for globus-url-copy $

10. In order to copy files from local machine to the server, issue a command like this (from the local machine):

$ globus-url-copy file:/home/<userid>/local_globus.txt \                   gsiftp://ldas-grid.ligo-la.caltech.edu/data2/ <userid>/local_globus.txt $ ssh ldas-grid.ligo-la.caltech.edu [...snip...] $ ls local_globus.txt  ... $ cat local_globus.txt This is a test file from the local machine for globus-url-copy $ exit $

11. globus-url-copy also allows for third party copies (i.e. directly from one remote server to anther remote server -- without relaying through the local machine).  This is done as follows:

This example uses hydra.phys.uwm.edu as the second GridFTP server.  Feel free to substitute any other GridFTP server on which you have a local account.

$ globus-url-copy gsiftp://ldas-grid.ligo-la.caltech.edu/data2/ <userid>/party_globus.txt \ gsiftp://hydra.phys.uwm.edu/home/ <userid>/party_globus.txt $ ssh hydra.phys.uwm.edu [...snip...] $ ls party_globus.txt  ... $ cat party_globus.txt This is a test file for 3rd party transfer using globus-url-copy $ exit $

Ask your instructor to demo this feature if you do not have access to a second GridFTP server.

12. Now let's simulate a problem.  Destroy your proxy certificate using grid-proxy-destroy.

$ grid-proxy-destroy

13. Now, try and copy a file from the server to your client machine.  You should see an error like the following:

$ globus-url-copy gsiftp://ldas-grid.ligo-la.caltech.edu/data2/ <userid>/party_globus.txt                   file:/home/<userid>/party_globus_again.txt error: globus_l_ftp_control_send_cmd_cb: gss_init_sec_context failed GSS failure: GSS Major Status: General failure GSS Minor Status Error Chain: init_sec_context.c:114: gss_init_sec_context: Error with gss context globus_i_gsi_gss_utils.c:308: globus_i_gsi_gss_create_and_fill_context: Error with GSI credential acquire_cred.c:125: gss_acquire_cred: Error with GSI credential globus_i_gsi_gss_utils.c:1310: globus_i_gsi_gss_cred_read: Error with gss credential handle globus_gsi_credential.c:721: globus_gsi_cred_read: Valid credentials could not be found in any of the possible locations specified by the credential search order. globus_gsi_credential.c:447: globus_gsi_cred_read: Error reading host credential globus_gsi_system_config.c:3977: globus_gsi_sysconfig_get_host_ cert_filename_unix: Error with certificate filename globus_gsi_system_config.c:380: globus_i_gsi_sysconfig_create_cert_string: Error with certificate filename: /etc/grid-security/hostcert.pem not owned by current user. globus_gsi_credential.c:239: globus_gsi_cred_read: Error reading proxy credential globus_gsi_system_config.c:4660: globus_gsi_sysconfig_get_ proxy_filename_unix: Could not find a valid proxy certificate file location globus_gsi_system_config.c:4657: globus_gsi_sysconfig_ get_proxy_filename_unix: File does not exist: Proxy file /tmp/x509up_u<uid> does not exist globus_gsi_credential.c:351: globus_gsi_cred_read: Error reading user credential globus_gsi_credential.c:1086: globus_gsi_cred_read_key: Key is password protected: GSI does not currently support password protected private keys. OpenSSL Error: pem_lib.c:434: in library: PEM routines, function PEM_do_header: bad password read

This error message is not that clear on what is wrong and why things failed. 

If you get messages like this always check to make sure you have a valid proxy certificate before doing anything else.

14. Create a new proxy certificate for yourself and try again.

$ grid-proxy-init Your identity: /C=US/O=National Center for Supercomputing Applications/CN=Mike Freemon Enter GRID pass phrase for this identity: Creating proxy ......................................... Done Your proxy is valid until: Wed Feb 23 03:47:37 2005 $ globus-url-copy gsiftp://ldas-grid.ligo-la.caltech.edu/data2/ <userid>/party_globus.txt \                   file:/home/<userid>/party_globus_again.txt

This time the copy should be successful.

UberFTP

15. Make sure you have access to UberFTP.

$ uberftp -v uberftp Version 1.3

To log on to a remote machine using UberFTP and GSI authentication.  To do this, we will need to use two command line options:

-H     host     Connect to host -a     auth     Use auth authentication. GSI and                 MSS (NCSA passwordless) authentication methods                 are supported. If this option is not used, the                 default is username/password

The -H option will specify the host that we want to logon to.  For example -H ldas-grid.ligo-la.caltech.edu.  The -a option tells uberftp what type of security authentication to use.  We want to use GSI, this allows us to access host using our proxy certificate.  This means, like with ssh, we will be able to access the host without entering our password.

16. Try the following:

$ uberftp -H ldas-grid.ligo-la.caltech.edu -a gsi 220 ligo-server.ncsa.uiuc.edu GridFTP Server 1.12 GSSAPI type Globus/GSI wu-2.6.2 (gcc32dbg, 1069715860-42) ready. 230 User mfreemon logged in.

17. You then get the "uberftp" prompt.

If you are behind a firewall, or the server cannot connect back to the client for whatever reason, then you need to switch UberFTP to passive mode using the passive command.

uberftp> passive passive

As an aside, there's nothing wrong with using passive mode all the time if you want to be in that habit.  In fact, starting with UberFTP version 1.6, UberFTP defaults to passive mode.

From this point on, UberFTP works just like regular FTP.

18. Look at the files found on the server:

uberftp> ls 150 Opening BINARY mode data connection. total 53 -rw------- 1 mfreemon 2383 Feb 22 15:50 .bash_history -rw-r--r-- 1 mfreemon  302 Feb 10 11:22 .bash_logout -rw-r--r-- 1 mfreemon  306 Feb 11 14:43 .bash_profile -rw-r--r-- 1 mfreemon  124 Feb 10 11:22 .bashrc -rw-r--r-- 1 mfreemon  383 Feb 10 11:22 .emacs drwxr-xr-x 3 mfreemon   80 Feb 21 13:00 .globus -rw-r--r-- 1 mfreemon  120 Feb 10 11:22 .gtkrc -rw-rw-r-- 1 mfreemon   41 Feb 22 14:50 local_scp.txt -rw-rw-r-- 1 mfreemon   41 Feb 22 14:37 server_scp.txt drwx------ 2 mfreemon   48 Feb 15 16:17 .ssh -rw-r--r-- 1 mfreemon  658 Feb 10 11:22 .zshrc 226 Transfer complete. uberftp>

19. Pick a file and transfer it from the server to your local machine.  You can do this by using the get command.  This command will get a file from the machine you have FTPed to and transfer it to the machine you FTPed from.

uberftp> ascii Transfer type set to ASCII uberftp> get server_scp.txt 150 Opening ASCII mode data connection. 226 Transfer complete. Transfer rate 45 bytes in 0.03 seconds. 1.32 KB/sec uberftp>

20. If you want to copy a file from your local machine to the remote host, simple use the put command.  This command is just like get, only it will take a file from your local machine and put it out to on the remote server.

uberftp> put local_scp.txt 150 Opening BINARY mode data connection. 226 Transfer complete. Transfer rate 47 bytes in 0.06 seconds. 0.75 KB/sec uberftp>

21. In order to exit from you UberFTP session just enter the command quit.

uberftp> quit 221-You have transferred 47 bytes in 1 files. 221-Total traffic for this session was 13550 bytes in 2 transfers. 221-Thank you for using the FTP service on ldas-grid.ligo-la.caltech.edu. 221 Goodbye.

22. Now we will use UberFTP to transfer a file between two remote hosts.  In this example we will start on our local client machine and transfer a file from ldas-grid.ligo-la.caltech.edu to hydra.phys.uwm.edu. 

Note:  If you do not have access to a second grid server (such as hydra.phys.uwm.edu), your instructor can demonstrate this feature of UberFTP during the lecture portion of the workshop.

You will be starting on your local machine.  From there, uberftp to ldas-grid.ligo-la.caltech.edu. 

$ uberftp -H ldas-grid.ligo-la.caltech.edu -a gsi 220 ldas-grid.ligo-la.caltech.edu GridFTP Server 1.12 GSSAPI type Globus/GSI wu-2.6.2 (gcc32dbg, 1069715860-42) ready. 230 User mfreemon logged in. uberftp>

23. Once there, use the lopen command to log onto hydra.phys.uwm.edu.

uberftp> lopen -H hydra.phys.uwm.edu -a gsi 220 hydra.phys.uwm.edu GridFTP Server 1.12 GSSAPI type Globus/GSI wu-2.6.2 (gcc32dbg, 1069715860-42) ready. 230 User mfreemon logged in. uberftp>

When connected like this, the ls command will show us the contents of the ldas-grid.ligo-la.caltech.edu server and lls will show us the files on  hydra.phys.uwm.edu.

24. The get command will take the file from ldas-grid.ligo-la.caltech.edu and put it on hydra.phys.uwm.edu

uberftp> get some_file_on_ldas-grid.txt src> 150 Opening BINARY mode data connection. dst> 150 Opening BINARY mode data connection. src> 226 Transfer complete. dst> 226 Transfer complete. uberftp>

25. To transfer a file from hydra.phys.uwm.edu to ldas-grid.ligo-la.caltech.edu use the put command.

uberftp> put some_file_on_hydra.txt dst> 150 Opening BINARY mode data connection. src> 150 Opening BINARY mode data connection. src> 226 Transfer complete. dst> 226 Transfer complete. uberftp>

26. To exit out of all your sessions type quit.

uberftp> quit 221-You have transferred 52 bytes in 2 files. 221-Total traffic for this session was 11390 bytes in 4 transfers. 221-Thank you for using the FTP service on hydra.phys.uwm.edu. 221 Goodbye. 221-You have transferred 50 bytes in 2 files. 221-Total traffic for this session was 15143 bytes in 4 transfers. 221-Thank you for using the FTP service on ldas-grid.ligo-la.caltech.edu. 221 Goodbye.