GridSAM-NGSSurgery-20090729

obtainablerabbiΔιαχείριση Δεδομένων

31 Ιαν 2013 (πριν από 4 χρόνια και 4 μήνες)

202 εμφανίσεις


Web: www.omii.ac.uk Email: info@omii.ac.uk

Overview of GridSAM and its
Deployment

Steve Crouch, OMII
-
UK

s.crouch@omii.ac.uk


29/07/09


Web: www.omii.ac.uk Email: info@omii.ac.uk

Contents


What is GridSAM and what does it offer?

o
For resource owners and end users


Example scenario


Packaging options


The job submission pipeline


Open development


Installation and configuration


Future developments


Web: www.omii.ac.uk Email: info@omii.ac.uk

GridSAM Overview


What is
GridSAM

to the resource owners?

o
A web service to uniformly expose a computational resource


Condor


Portable Batch Scheduler (PBS)


Globus


Sun
GridEngine


Platform Load Sharing Facility (LSF)


Single machine through Fork or SSH

o
Acts as a client to these resources


What is
GridSAM

to end
-
users?

o
A means to access computational resources in an open standards
-
based
uniform way

o
A set of end
-
user command
-
line tools and client
-
side APIs to interact
with
GridSAM

Web Services


Submit and monitor compute jobs


Cross
-
protocol file transfer (
gsiftp
, ftp,
sftp
,
WebDav
, http, https, soon SRB,
iRODS
) via Commons
-
VFS (
http://sourceforge.net/projects/commonsvfsgrid
)



Web: www.omii.ac.uk Email: info@omii.ac.uk

For Resource Owners…


Computational

Resource

Manager

DRM



One of:

PBS (Torque/OpenPBS/PBSPro)

LSF, Condor, Sun GridEngine,
Globus, Fork

GridSAM

Service

X509

certificate

Linux Many flavours:

RHEL 3,4,5, Fedora 7,8,
Scientific Linux 4

Java: JDK 1.5.0+

Linux +
Java

Tomcat/

Axis

Tomcat: 5.0.23, 5.0.28, 5.5.23

Axis: v1.2.1

Persistence provided by one of:

Hypersonic, PostgreSQL, or
existing MySQL


Web: www.omii.ac.uk Email: info@omii.ac.uk

For End
-
Users…


GridSAM

Service

JSDL

HTTPS/HTTP

WS
-
Security:

X509

User/Password

Service
Interface

Any/all of:

GridSAM native interface, OGSA
-
BES v1.0, HPC Basic Profile v1.0



MyProxy

(for
Globus
/

GridFTP
)

+ MyProxy

credentials

Windows/

Linux +
Java

GridSAM

Client

Many flavours:

RHEL 3,4,5, Fedora 7,8,
Debian, Ubuntu, Scientific
Linux 4, Windows XP,
Windows Vista

Java: JDK 1.5.0+

Axis

X509

certificate

Generic
BES/HPC
Basic Profile

Client

OSGA
-
BES

HPC Basic
Profile


Web: www.omii.ac.uk Email: info@omii.ac.uk

Example Scenario
-

CPOSS ENGAGE Project

Condor



UCL Legion

(PBS Torque
+ Moab)



UK NGS



Job
Management

System

Workflow

Portal

GridSAM

GridSAM

Active

BPEL



Web: www.omii.ac.uk Email: info@omii.ac.uk

Package Options: CGT


Campus Grid Toolkit (CGT)


automated client or server install


Client/server solution that contains

o
Apache Tomcat/Axis/WSS4j (WS
-
Security)

o
GridSAM

o
Optional
-

Application Hosting Environment (AHE)

o
PostgreSQL

or
HyperSonic

database


Easy, quick installation via further
autoconfiguration
:

o
Service:


Initial security setup for testing


Hooks into existing computational resources

o
Client:


Downloadable from existing CGT server and pre
-
configured for that server


End
-
to
-
end, ‘out of the box’ working examples


Currently version 1.1.3 (with
GridSAM

2.1.6)


Installation videos


see
http://www.omii.ac.uk/wiki/CGT


Web: www.omii.ac.uk Email: info@omii.ac.uk

Package Options: Development Kit


Development Kit


heavily assisted client or server installation


Service comprised of:

o
Web Services Container:
Tomcat/Axis with optional WS
-
Security

o
Data persistence: readily
-
installable
PostgreSQL

or
HyperSonic

databases, or pre
-
installed
MySQL

o
Configurable Distributed Resource Manager (DRM) system


Client essentially ‘container’ of libraries and security
configuration


Optionally downloadable and installable components for
client and server:

o
GridSAM
: Job submission and monitoring

o
AHE: Application Hosting Environment

o
Grimoires
: UDDI + metadata registry

o



Currently version 3.4.4



Web: www.omii.ac.uk Email: info@omii.ac.uk

GridSAM e.g. with Condor


A staged event
-
driven architecture

o
Submission pipeline is constructed as a
network of stages

connected by
event queues

o
Each stage performs a specific action upon incoming events

Example Pipeline: Condor


Web: www.omii.ac.uk Email: info@omii.ac.uk

Open Community Development


GridSAM is Open Source,
Open Community
Development


GridSAM SourceForge project:

o
99.03% activity, 1 release/month

o
SVN source code repository

o
Developer & discuss mailing lists


http://sourceforge.net
/projects/gridsam/


Web: www.omii.ac.uk Email: info@omii.ac.uk

Installation Process


e.g. Development Kit


Pre
-
requisites:

o
Download
omii
-
server
-
3.4.4.tar

from OMII
-
UK website (Software
-
>
Downloads) & unpack

o
Ensure JDK 1.5.0+ is installed & JAVA_HOME set & Java on PATH

o
e.g. for PBS


Ensure submit and monitor PBS commands available on PATH


Installable on PBS server node, or elsewhere with
ssh

access to PBS server node


cd

to omii
-
server
-
3.4.4 unpacked directory and run
./
OMIIstackInstall.pl

once each for:

1.
Database install: provide
PostgreSQL

version, install location & port

2.
Tomcat/Axis container: Tomcat version, install location, database port,
test certificate details

3.
Database setup for optional components

4.
Optional components: select
GridSAM

only


Install complete



Web: www.omii.ac.uk Email: info@omii.ac.uk

Config. Process


e.g. Development Kit


For example, for PBS (any flavour):

o
GridSAM

is a PBS client for submission and monitoring

o
Edit
usermap_config.xml

and supply DN
-
> local user
account mappings

o
Edit
batch_config.xml

for local PBS configuration


PBS batch queue name to use


User with
sudo

privileges to enable job submission as submitting
user

o
Ensure PBS
qstat
,
pbsnodes
,
qnodes
,
qsub

on PATH when
starting container

o
Restart container


For CGT, you specify install requirements in an
easily changed specification file prior to installation


Web: www.omii.ac.uk Email: info@omii.ac.uk

Planned Future Developments


For end
-
users:

o
Refactored

documentation and improved OGF standards
coverage

o
Full support for JSDL
DataStaging

credential specification

o
Full support for JSDL Resource selection across PBS, Condor &
Fork
DRMs

o
JSDL Parameter Sweep Extension

o
Support for SRB and
iRODS


For resource owners:

o
Packaging option as a standalone, manually configurable web
archive (WAR) file

o
Job submission through multiple remote
ssh

accounts


Direct PBS deployment throughout NGS sites