POSIX-like OGSA/SOAP Services

clappingknaveΛογισμικό & κατασκευή λογ/κού

14 Δεκ 2013 (πριν από 3 χρόνια και 5 μήνες)

99 εμφανίσεις

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

POSIX
-
like OGSA/SOAP Services

Arun Jagatheesan

Architect & Team Lead, SDSC Matrix

San Diego Supercomputer Center

GFS, Global Grid Forum
-
9

October 7, 2003, Chicago

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

2

Talk Outline


Grid File System


The small big picture


Need for Schema


Need for Operation definitions


Data Transport

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

3

Grid File System

Data Sources

Grid File System Service (POSIX
-
like Interface)

Data Services

Virtual Directory Service

(Management of virtualization)

Coordinated with
other groups

Hierarchical
Logical
Name space,
ACL,
metadata

Applications (Astronomy, Physics, Life Science, business apps, . . .)

NFS/CIFS …

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

4

The small big picture

Data Sources

Grid File System Service (POSIX
-
like Interface)

Data Services

Virtual Directory Service

(Management of virtualization)

NFS/CIFS …

XML
Schema for
Collections
, Data Sets

OGSA/SOAP
based
interfaces for
file operations

NFS or other
standard
interface over
the virtualized
schema

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

5

Grid Collection Schema


XML Schema based Description for


Collections or Virtual Directories


Data Sets


File System Meta
-
data (file size, date created, …)


Application Specific Meta
-
data


Access Permissions






Logical Name space


Extensible


Scalable (more federations)


Dynamic Composition of the name space


Import and Export

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

6

Operations on Logical Namespace


OGSA/SOAP based interfaces


Grid File System operations


Similar to traditional file systems operations / POSIX


Open (= Get a GSR?), Read, Seek’n’Read, Seek’n’Write, …


Simple Control (Context) Operations


Management of Logical Namespace


SOAP based bindings


Bulk (Content) Operations


Only SOAP bindings for data transport ??? (NOPE)


Alternative mechanisms needed in standard


San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

7

How do we form the logical
namespace?

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

8

Logical Layers (bits,data,information,..)

Storage Resource Transparency

Storage Location Transparency

E:
\
srbVault
\
image.jpg

/users/srbVault/image.jpg

Select … from srb.mdas.td where...

Data Identifier Transparency

image_0.jpg…image_100.jpg

Data Replica Transparency

image.sql

image.cgi

image.wsdl

Virtual Data Transparency

Collections or Virtual Directories

patientRecordsCollection

myActiveNeuroCollection

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

9

Storage Resource Transparency (1)


Storage repository abstraction


Archival systems, file systems, databases, FTP sites, …


Logical resources


Combine physical resources into a logical set of resources


Hide the type and protocol of physical storage system


Load balancing


based on access patterns


Unlike DBMS, user is aware of logical resources


Flexibility to changes in mass storage technology

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

10

Storage Resource Transparency (2)


Standard operations at storage repositories


POSIX like operations on all resources


Storage specific operations


Databases
-

bulk metadata access


Object ring buffers
-

object based access


Hierarchical resource managers
-

status and staging
requests

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

11

Storage Location Transparency


Support replication of data for performance


Transparent access to physical location and physical
resource


Virtualization of distributed data resources


Data naming managed by the data grid


Redundancy for preservation


Resource redundancy


“m of n” resources in list


Location redundancy


replicate at multiple locations

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

12

Data Identifier Transparency


Four Types of Data Identifiers:

1.
Unique name


OID or handle

2.
Descriptive name


Descriptive attributes


meta data


Semantic access to data

3.
Collective name


Logical name space of a collection of data sets


Location independent

4.
Physical name


Physical location of resource and physical path of data

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

13

Data Replica Transparency


Replication


Improve access time


Improve reliability


Provide disaster backup and preservation


Physically or Semantically equivalent replicas


Replica consistency


Synchronization across replicas on writes


Updates might use “m of n” or any other policy


Distributed locking across multiple sites


Versions of files


Time
-
annotated snapshots of data

San Diego Supercomputer Center

National Partnership for Advanced Computational Infrastructure

University of Florida

14

Conclusion


Lot of possibilities


Need for a Standard Grid File Schema and
Global Logical Namespace for virtualization


Need for Standard description of Operations or
Grid File System Service


Call for


Users, Projects


Developers, Vendors


It’s a stone’s throw away


together, we will do it.