myproxy-retrieve - cyberaide

arghtalentΔιαχείριση Δεδομένων

31 Ιαν 2013 (πριν από 4 χρόνια και 5 μήνες)

280 εμφανίσεις

?

Rochester Institute of Technology

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

1

Globus Toolkit 4

By:

Jeton Bacaj, Joel Lathrop, Fugang
Wang
? &
Andrew Younge


http://blackrose02.rit.edu/wiki/doku.php?id=grid:seminar1:gt4

?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

2

Outline


Introduction


Components


Execution Management


Information Services


Data Management


Security


Globus Implementation


Architectural overview


Globus and Rocks


Todo


Job submission


Summary



?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

3

Introduction to Globus Toolkit


What is Globus?


A “grid” toolkit


The goal is,
“Letting people share computing power, databases,
and other tools securely online across corporate, institutional,
and geographic boundaries without sacrificing local autonomy.”


Modular Architecture


Made of many components


Not all components are inter
-
dependent


Can install single, isolated components


Globus Advantages


Scalable


Well documented API


Open source


Widely used

?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

4

Introduction (cont)


Components organized into classes


Execution Management


Initiation, monitoring, management, scheduling, and coordination
of remote computational tasks


Information Services


Monitoring, managing logs, archiving data, etc.


Data Management


Data location, transfer, and management


Security


Authentication, mapping Globus credentials with other forms of
credentials


Common Runtime


A collection of APIs allowing integration with different languages
and platforms


?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

5

Execution Management


Community Scheduler Framework


Interface to resource schedulers


Grid TeleControl Protocol


WRSF
-
enabled service interface for control of remote
instruments


Workspace Management


Dynamic fostering of managing workspaces on hosts


Grid Resource Allocation & Management


“GRAM”; the bread ‘n butter of Globus Toolkit’s Execution
Management


deploying and monitoring jobs on a Grid


Used to incorporate a wide variety of other grid and cluster services such as
Condor, PBS, LSF, SGE, BOINC and more!


?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

6

Information Services

Some of the services that are included on Globus Toolkit:


MDS


Collects data about different resources about the grid


By default, uses WSRF for polling operations


WebMDS uses MDS that formats data in XML into HTML using XSLT
style sheets


Indexing


Services register with this component so others can utilize those
resources


Caches resources of the registered services (speed obviously)


Trigger


What it sounds like
-

if there is a condition that’s met, an alert is raised
(notification via email for example)


Monitors behavior
-

looks at the data with some preconfigured tests to
see if trigger conditions are met


?

Data Management and Replication


Data Management


GridFTP


A Souped up version of FTP that’s secure, robust and
efficient


Good for bulk data.


Reliable File Tranfter (RFT)


More reliable, job based file transfer using web services


Data Replication


Replica Location Service (RLS)


Keeps track of location of the copies in the grid


Data Replication Service (DRS)


Higher level merge of RFT and RLS


02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

7

?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

8

Grid Security Infrastructure
-

GSI



Conform to X.509 based PKI standard


Certificate


Used to identify and authenticate


Structure


(subjID, publicKey, CA’s ID, CA’s signature)


Coded in X.509 format


Host, service, user, proxy


Mutual authentication


Use SSL/TLS to exchange certificates


Prove identity by signing a message with private key


?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

9

Grid Security Infrastructure
-

GSI



Confidential communication


By default no encryption


Easy to setup a shared key encryption


Securing private key


Pass phrase protected private key


Proxy and certificate delegation


Chain of trust


RFC 3820


?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

10

GSI
-

MyProxy


MyProxy credential management service



User_home

MyProxy

Server

Grid

Front End

CA

?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

11

GSI
-

MyProxy


MyProxy credential management service



MyProxy

Server

Grid

Front End

User_mobile

?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

12

MyProxy Commands


MyProxy related commands in GT4



myproxy
-
init
: store proxy credential

myproxy
-
store
: store end
-
entity credential

myproxy
-
logon
: retrieve a stored credential

myproxy
-
retrieve
: retrieve end
-
entity credential


myproxy
-
destroy
: remove credential

myproxy
-
info
: query stored credentials



myproxy
-
change
-
pass
-
phrase
:

change password encrypting private key


?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

13

GSI
-

In Practice


User certificate generation



User generates certificate request and private key

Private key(passphrase protected):


~/.globus/userkey.pem

Certificate request:


~/.globus/usercert_request.pem

USER$ echo export GLOBUS_LOCATION=/usr/local/globus
-
4.0.6 >> ~/.bashrc

USER$ $GLOBUS_LOCATION/etc/globus
-
user
-
env.sh

USER$ grid
-
cert
-
request

! Mail the certificate request to the administrator as prompted

?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

14

GSI
-

In Practice


User certificate generation



CA signs the certificate

User generates certificate request and private key

globus$ grid
-
ca
-
sign

in usercert
-
request.pem
-
out signed_USER.pem

globus$ sudo cp signed_USER.pem /home/USER/.globus/usercert.pem


?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

15

GSI
-

In Practice


User certificate generation



CA signs the certificate

Add entry in gridmap file

User generates certificate request and private key

"/O=Grid/OU=GlobusTest/OU=simpleca
-
lilac03.rit.edu/OU=rit.edu

/CN=FIRST_NAME LAST_NAME" USER

/etc/grid
-
security/grid
-
mapfile

?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

16

Grid Architecture

?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

17

Progress


Main Globus installation


Used Ubuntu Linux 7.10 Server


Compiled and installed version 4.0.6 of Globus


Setup security using Simple
-
CA


Configured postgresql RFT Database


Started Globus container


Added test users and submitted test jobs


Rocks Cluster


Installed the frontend on lilac04.rit.edu


Added Globus, Ganglia, and SGE roles to the frontend


Installed clients on other lilac machines.


?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

18

Todo List


Link Rocks frontend to main Globus installation


Configure the Certificates


Setup and test GridFTP between machines


Update /etc/grid
-
security/grid
-
mapfile


Test the grid using by submitting a job to a
remote resource (Rocks)


Setup independent CA behind a firewall using
MyProxy


Would enhance security and simplify usage


?

Submitting a Job


Submitting a simple job


xero@lilac03:~$ globusrun
-
ws
-
submit
-
c /bin/echo ROAR


Submitting job...Done.


Job ID: uuid:a07e6908
-
fbb8
-
11dc
-
884b
-
000874d2a480


Termination time: 03/28/2008 04:45 GMT


Current job state: Active


Current job state: CleanUp


Current job state: Done


Destroying job...Done.


Job submission in Globus gets more complicated quickly


Have to use large xml (RSL) files to specify job details


Use a job_epr.xml file to monitor job status


Make sure job results are retrieved

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

19

?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

20

Summary


The Globus Toolkit is a powerful middleware
application that allows one to build a multi
-
level grid
computing system that can span spatially and
administratively different computing systems.


Setting up and managing a grid using Globus is a lot
of work!


Globus can work with a wide variety of clusters,
queuing systems or other grid technologies in order
to provide the largest amount of resources


System integration is unmatched by anything else


Many tools already available for free


Easy API to create custom tools for specific needs


?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

21

Questions?


Ask, and ye shall receive …

?

02:59:09

Service Oriented Cyberinfrastructure Lab, http://blackrose02.rit.edu

22

References

Foster, "A Globus Primer: Describing Globus Toolkit 4," 2005.

I. Foster, "Globus Toolkit 4: Software for Service
-
Oriented Systems," in Proc. Network And
Parallel Computing: IFIP International Conference, 2005, pp. 2
-
13.

C. K. Ian Foster, "Globus: a Metacomputing Infrastructure Toolkit," Iternational Journal of High
Performance Computing Applications, vol. 11, iss. 2, pp. 115
-
128, 1997.

L. P. N. M. C. K. I. F. M. D. Jennifer M. Schopf Ioan Raicu, "Monitoring and Discovery in a Web
Services Framework: Functionality and Performance of Globus Toolkit MDS4,"
Argonne National Laboratory2005.

W. Allcock, J. Bresnahan, R. Kettimuthu, and M. Link, "The Globus Striped GridFTP Framework
and Server," in Proc. SC '05: Proceedings of the 2005 ACM/IEEE conference on
Supercomputing, Washington, DC, USA, 2005, p. 54.

H. Kanemitsu and Y. Urano, "On the method for realizing globally and locally accessible
resource management for WS
-
GRAM," in Proc. Web Services, 2007. ICWS 2007.
IEEE International Conference on, 2007, pp. 1201
-
1204.

X. Zhang and J. M. Schopf, Performance Analysis of the Globus Toolkit Monitoring and
Discovery Service, MDS2, 2004.

I. N. Borja Sotomayor Lisa Childers, Globus Toolkit 4 Programming Java Services, Morgan
Kaufmann, 2006.