Data Grid Web Services

abnormalobeisanceΑσφάλεια

3 Νοε 2013 (πριν από 3 χρόνια και 7 μήνες)

74 εμφανίσεις

Data Grid Web Services

Chip Watson

Jie Chen, Ying Chen,

Bryan Hess, Walt Akers

A Three Tier Web Services Architecture

Web Browser

XML to HTML servlet

Web Service

Application

Web Service

Web Service

Grid Service

Local Backend
Services

(batch, file, etc.)

Web Server

(Portal)

Authenticated connections

Remote Web
Server

Web Service

Storage system

Grid resources,
e.g. Condor

Batch system

Why Web Services?

Strong industry support & growing adoption

Self describing interfaces & protocol

Support in all languages

Easy addition of additional input or output
parameters

Interface evolution w/o breaking what works

PPDG Architecture

Application
Planner
Executor
Catalog Services
Info Services
Policy/Security
Monitoring
Repl
. Mgmt.
Reliable Transfer
Service
Compute Resource
Storage Resource
DAG
DAG
Network Resource
PPDG Architecture

Application
Planner
Executor
Catalog Services
Info Services
Policy/Security
Monitoring
Repl
. Mgmt.
Reliable Transfer
Service
Compute Resource
Storage Resource
DAG
DAG
Network Resource
File Client

Meta Data Catalog

Replica Catalog

HRM++ Service

Replication Service

Storage Resource

File Server(s)

HRM Listener

Web Services

Single Site

Data Grid Web Services Architecture

Components: Replica Catalog

Get replicas: GFN
-
> SURLs

Get best replica?

Create replica

Input: GFN, SURL

Specify <meta data> for new GFN

Remove replica

Input: GFN, SURL

Make / delete directory (recursive)

Directory Listing

terse or verbose, optionally more than 1 level deep

optionally matching a pattern (regexp?)

Create / delete link (soft) to another file or directory

File Client

Meta Data Catalog

Replica Catalog

HRM++ Service

Replication Service

Storage Resource

File Server(s)

HRM Listener

Web Services

Single Site

Components: HRM Listener


This component serves as the link
between the grid
-
unaware HRM and
the replica system. The HRM /
storage resource generates 2
possible types of events.

Advice request:

proposed deletion of file X. Listener responds
with advice as a number in the range of 0.0 (please don’t) to 1.0
(OK). The listener could base this advice upon interaction with
the replica catalog to discover if this is the last disk resident
copy, for example.

State change notification:

File X is added, or deleted, or cache
state is changed. In this case the listener updates the replica
catalog.


File Client

Meta Data Catalog

Replica Catalog

HRM++ Service

Replication Service

Storage Resource

File Server(s)

HRM Listener

Web Services

Single Site

Components: Replication Service

This component acts as an agent

for the client to make replicas,

and manipulate replica policy




Web services:


Copy a replica of GFN / SURL to site X.

Get status of replication operation.

Add / edit / remove a local replication policy
(push, maybe pull)


To implement a replication policy, it may
register as a listener with the HRM

File Client

Meta Data Catalog

Replica Catalog

HRM++ Service

Replication Service

Storage Resource

File Server(s)

HRM Listener

Web Services

Single Site

Components: HRM++ Service

HRM Web Services:

File status (cached, pinned, permanent, size, owner, etc.)

File status changes (e.g. stage a file, pin a file, make permanent)

Mapping from SURL to TURL for file get, including protocol
negotiation

Space allocations for put, including


protocol negotiation to yield TURL



Extended functions:

Directory listings, search (like replica catalog)

Reliable (as much as possible) third party file transfers to/from
another Data Grid Site (reliable), or to/from a site with a supported
protocol (e.g. ftp site)

File Client

Meta Data Catalog

Replica Catalog

HRM++ Service

Replication Service

Storage Resource

File Server(s)

HRM Listener

Web Services

Single Site

Technologies Employed

Apache web server

Tomcat servlet engine

JAXM for SOAP Messages

XML data format

File Client

Meta Data Catalog

Replica Catalog

HRM++ Service

Replication Service

Storage Resource

File Server(s)

HRM Listener

Web Services

Single Site

Implementation

Replica Catalog

SOAP servlet + mySQL back end

(future) global replication policy, client to replication service

HRM++ Service

HRM: SOAP servlet wrapping JASMine

Extensions to HRM:


reliable file transfer (wrap gridftp, etc.), queuing


directory listings, tree search

Replication Service

SOAP servlet + mySQL for request persistence & queues

(future) listener for new files + policy for replication (push)

HRM Listener

SOAP servlet, client to Replica Catalog

Status

Year old raw XML limited prototypes:

Replica catalog


Read
-
only listings, GFN
-
> SURL


Loaded with silo info (>100,000 files)

Pre
-
HRM service


Read
-
only listings, SURL
-
> TURL (multi
-
protocol)


New SOAP components currently in development

Replica catalog


full capabilities except ACL’s, user defined meta
-
data (deferred)

HRM++ service


Recursive file transfer client <
-
> unmanaged storage (jparss)


3
rd

party reliable file transfers

WSDL

Web Services Definition Language


(equivalent to CORBA IDL)



http://lqcd.jlab.org/grid/gridService_wsdl.xml


Data Grid File Manager
Client Application

Capabilities (prototype)

Browse

contents of file system

Managed disk cache on data grid node

Unmanaged Local or Remote file system

Tertiary storage (eventually HRM)

Move

files between managed and unmanaged storage

Within a single data grid node

Between local file system and data grid node

1Q02: Between data grid nodes (3
rd

party transfer)

Status



displays if file is currently in disk cache

Migrate

from tape to disk (not released)

Standardization Activities

PPDG Activity:


Jlab is working with the SRB group to
standardize web services (WSDL) for
managing a data grid

Common interface for JASMine and SRB

Web services client to inter
-
operate between
dissimilar back ends

Extend to additional systems once operational