Data Grid Web Service

hurtpotatocreekSecurity

Nov 5, 2013 (3 years and 1 month ago)

48 views

JLab Lattice Portal



Data Grid Web Service

Ying Chen, Chip Watson

Thomas Jefferson National Accelerator Facility

Jlab Lattice Portal


Jefferson Lab (JLab) and its collaborators at
MIT are in the process of developing a web
portal for the Lattice Hadron Physics
Collaboration.



This portal will eventually provide access to
Linux clusters, disk caches, and tertiary
storage located at JLab, MIT, and other
universities.

Jlab Lattice Portal

Jlab Lattice Web Portal infrastructure capabilities:


-

Publish/locate a data set.


-

Submit/monitor/control a batch job.


-

Move a data set to/from the compute resource.


-

Authenticate/authorize and track the use of the
resources.


-

...


Why Grid Web Service ?

Disk Mgr

Tape Mgr

users

Managed disk

Unmanaged disk

tertiary storage

users

users

login

login

login

Data Grid
Node

Fire wall

Data Grid Web Service


Hides the complexity of the underlying
system


users don’t have to deal with the disk manager or tape
manager etc.


Provides the service on the web


users don’t have to login to do the work

System Architecture Overview



Grid Service

Disk Mgr

Tape Mgr

users

Managed disk

Unmanaged disk

tertiary storage

users

users

Web
brow
ser


web
browser

Fire wall

Web Based Grid Service


Jlab Data Grid Web Service
-

a java
servlet


-

uses apache with model
-
ssl as secure web
server and tomcat as servlet engine


-

service request is an xml format string


-

the result/response is also in xml format



-

a certificate is required for the privileged
services

Service Request/Response

Request:



<info name="version“ />

Response:

<grid name="LQCD at Jlab“ url="lqcd.jlab.org/servlet/grid.data.Service">


<info name="version">



<info>jlab data grid version '0.1'</info>


</info>

</grid>


----------------------

or
-------------------

<grid name=“LQCD at Jlab” url=“lqcd.jlab.org/servlet/grid.data.Service”>


<info nam=“version”>



<error>Attribute “name” missing</error>


</info>

</grid>

Why xml ?


Self
-
describe (easy to understand)


Language independent


easy to assemble from multiple source


Interchange of data among application


Focus upon the data structure
(can easily
convert to different presentation)


Many free or commercial tools available to
process xml data

http://lqcd.jlab.org/pbs/

Jlab Web Based Grid Service


The grid service includes:


-

provides node information and status


-

lists the contents of the grid node


-

moving files within the node


-

performs file operations
(delete, pin/unpin,
mkdir, mkdirs etc.)



The URL of the service:

http://lqcd.jlab.org/servlet/grid.data.GridService

Use the Service
-

directly

Enter the url in a web browser:

http://lqcd.jlab.org/servlet/grid.data.GridService
?service=<list+name='$CACHE/ychen'/>


or

https://lqcd.jlab.org/servlet/grid.data.GridService
?service=<file
-
operation+type=‘delete’+
name=‘$DATA/foo'/>


Use the Service
-

Grid API


A
grid i
n
terface API
is designed for using
this Data Grid web service.


A java implementation of this API is being
developed to encapsulate this xml
service.


Using this API to obtain the service, the
xml input and output are hidden by the
implementation.

Grid API
-

GridFactory


public class GridFactory


provides an
interface to access the GridNode object
using static methods.





public static

GridNode getGridNode (String url)




public static

GridNode getGridNode ()



public static

GridNode getReplica ()


interface GridNode


Is
a grid node or
replica catalog object that can be obtained
from the GridFactory class.

Grid API
-

GridNode

public interface GridNode {


public String getURL ();


public String getName();


public GridDirectory getDirectory (String path);


public GridFile[] list(String path);


public String[] listNames (String path);


public boolean isExist (String path);


public boolean isDirectory (String path);



public boolean isFile (String path);

}

Use the Grid API

public class GridAPIDemo {


public static void main (String[] args) {


try {



GridNode jlab = GridFactory.getGridNode();




NodeDirectory dir = (NodeDirectory) jlab.getDirectory(args[0]);



String[] names = dir.listNames();



System.out.println(args[0]+“ directory contains:”);



for (int i=0; i<names.length; i++)




System.out.println(names[i]);


} catch (Exception e) { System.out.println(e.getMessage()); }


}

}

Future Developments


Standardize the xml request and response.


Add additional web technology (SOUP,
WSDL, and UDDI) to build dynamic web
-
based system


Extend the batch web interface/service.


Add third party file transfer capability using
remote web services (web server to web
server)

Questions ?