MOCCA - H2O-based CCA component framework for programming grids and metacomputing systems

currygeckoΛογισμικό & κατασκευή λογ/κού

2 Δεκ 2013 (πριν από 3 χρόνια και 4 μήνες)

275 εμφανίσεις

Institute of Computer Science
AGH

MOCCA
-

H2O
-
based CCA component
framework for programming

grids and
metacomputing systems

Maciej Malawski,
Marian Bubak,

Michał Placek, Daniel Harężlak,

Dawid Kurzyniec,

Vaidy Sunderam


Institute of Computer Science AGH, Krak
ó
w, Poland

Academic Computer Centre CYFRONET
-
AGH, Kraków, Poland

Distributed Computing Laboratory in the Dept. of Math and Computer
Science, Emory University, Atlanta


Institute of Computer Science
AGH

2

Outline


Motivation


programming grids


Component approach

and
CCA as a
component standard


H2O resource sharing platform


Design and implementation of MOCCA


Applications and i
nitial results


Future research



GCM interoperability



Institute of Computer Science
AGH

3

Problem


how to program the
Grid


Many programming models:


MPI


Service Oriented Architectures, Web Services


Tuple spaces, HLA


Distributed Objects


Custom protocols


Components


...


Institute of Computer Science
AGH

4

Common Component
Architecture


Component standard for High Performance
Computing


Uses and provides ports described in SIDL


Support for scientific data types

(complex
numbers, data arrays)


Existing tightly coupled (CCAFFEINE) and loosely
coupled, distributed (XCAT) frameworks

Institute of Computer Science
AGH

5

Existing CCA Frameworks


CCAFFEINE


Tightly coupled


Support for Babel


MPI support


XCAT


Loosely coupled


Globus
-
compatible


Java
-
based


DCA


MPI based


MxN problems


SCIRun2


Metacomponent model


LegionCCA


Based on Legion Metacomputing system

Institute of Computer Science
AGH

6

Requirements for a
Metacomputing
-
oriented CCA
Framework


Facilitated deployment

-

provide easy mechanisms for
creation of components on distributed shared resources;


Efficient communication

-

both for distributed and local
components;


Flexible

-

allow flexible configuration of components and
various application scenarios;


Support native components
, i.e. components written in
non
-
Java programming languages and compiled for specific
architecture.


Interoperable

with Grid standards (Web services)


Solution: H2O

Institute of Computer Science
AGH

7

H2O Resource sharing
platform


Providers own resources


They
independently

share
them over the network


access control policies


Clients discover, locate,
and utilize resources


Resources
configurable

via
plugins


Aggregation and reselling:

cascading pairwise
relationships

Institute of Computer Science
AGH

8

H2O Component Model


Nomenclature


container =
kernel


component =
pluglet


Pluglet = remotely accessible object


implements
Pluglet

interface, used by
kernel to signal/trigger pluglet state
changes






Remote access: based on the RMI model


Pluglets export
functional
remote interfaces

Pluglet

Pluglet

Functional

interfaces

Kernel

Clients

(e.g.
Hello
)

tutorial/step1/srv/Hello.java

public interface
Hello
extends
Remote
{


String hello() throws RemoteException;

}

Interface Pluglet {


void init(RuntimeContext cxt);


void start();


void stop();


void destroy();

}

Institute of Computer Science
AGH

9

Example scenarios of H2O


1. Provider = deployer



e.g. resource = legacy
application

2. Reseller:=

developer = deployer



e.g. computational service
offered within a grid system

3. Client = deployer



e.g. client runs custom
distributed application on
shared resources

Deploy

B

A

Legacy

App

Deploy

Provider

A

Client

Repositor
y

A

B

Reseller

C

Deploy

A

native

code

Provider

Client

Repositor
y

A

B

Developer

C

Provider

Client

B

A

Registration and Discovery

e
-
mail,

phone, ...

JNDI

UDDI

LDAP

DNS

GIS

...

B

Publish

Find

Provider

Institute of Computer Science
AGH

10

RMIX Communication
Substrate


Extensible framework


Remote Method Invocations
paradigm


Pluggable protocol providers


Multiple protocols supported


JRMPX, ONC
-
RPC, SOAP


Request
-
Response and
Asynchronous calls


Combines simplicity, flexibility,
and performance

ONC
-
RPC

Web Services

SOAP clients

JXTA

RMIX

RMIX

XSOAP

RMIX

RPCX

RMIX

JXTA

RMIX

JRMPX

Java

Service

Institute of Computer Science
AGH

11

RMIX: multiple protocols


Protocol switching


Protocol negotiation


Various protocol stacks for
different situations


SOAP: interoperability


SSL: security


ARPC, custom (Myrinet,
Quadrics): efficiency


Harness

Kernel

Internet

security

firewall

efficiency

efficiency

H2O Kernel

H2O Kernel

H2O Kernel

H2O Kernel

H2O Kernel

Institute of Computer Science
AGH

12

RMIX over JXTA


Fully operational RMI implementation running over
JXTA P2P network


Methods can be
invoked on remote
objects located
behind firewalls or
NATs


Our implementation
of JXTA socket
factories manages
all the JXTA
connectivity
transparently from
user
’s

point of view

Institute of Computer Science
AGH

13

MOCCA Implementation in H2O


Each component running in separate pluglet


Facilitated deployment and security


Thanks to H2O kernel security mechanisms, multiple components
may run without interfering


Using RMIX for communication


efficiency, multiprotocol
interoperability


Flexibility and multiple scenarios


as in H2O


MOCCA_Light: pure Java implementation
-

need for supporting
multilanguage components


Institute of Computer Science
AGH

14

Remote Port Call

Component Pluglet
CCA
Component
MOCCA
Services
1. getPort
2. create
3. call
Component Pluglet
CCA
Component
Dynamic
Port
Proxy
4. RMIX call
5. call
User Side
Provider Side
Institute of Computer Science
AGH

15

How to use MOCCA

(step by step)


Implement component code extending CCA interfaces
(cca.Port, cca.Component)


Compile component classes into JAR file


Publish application JARs on HTTP server


Use the Java client API or write a Jython script to assemble
application from components


Specify components and their connections


Specify locations of H2O kernels where to instantiate
components


Running the script automatically deploys necessary pluglets
into H2O kernels and spawns application

Institute of Computer Science
AGH

16

Example script

builder = MoccaMainBuilder()

uriKernel1 = URI.create("
http://emily.mathcs.emory.edu:7800/
")

uriKernel2 = URI.create("
http://zeus10.cyf
-
kr.edu.pl:7800/
")

userBuilderID = builder.addNewBuilder(uriKernel1, "
MyBuilderPlugletA
")

providerBuilderID = builder.addNewBuilder(uriKernel2, "
MyBuilderPlugletB
")

properties = MoccaTypeMap()

properties.putString("
mocca.plugletclasspath
",


"
http://emily.mathcs.emory.edu/mocca/mocca
-
samples.jar
")

properties.putString("
mocca.builderID
", userBuilderID.getSerialization())

userID = builder.createInstance("
My StarterComponent
",


"
mocca.samples.pingpong.impl.MoccaStarterComponent
”,


properties)

properties.putString("
mocca.plugletclasspath
",


"
http://emily.mathcs.emory.edu/mocca/mocca
-
samples.jar
")

properties.putString("
mocca.builderID
", providerBuilderID.getSerialization())

providerID = builder.createInstance("
MyPingComponent
",


"
mocca.samples.pingpong.impl.PingPongComponent
",


properties)

connectionID = builder.connect(userID,


"
PingPongUsesPort
",


providerID,


"
PingPongProvidesPort
")

MoccaBuilderClient.invokeGo(userID)


Institute of Computer Science
AGH

17

Automatic Flow Composer
Example

Lookup
Flow
Optimizer
Flow
Composer
Link
Evaluator
Site
Evaluator
Component
Registry
Evaluate
Compose
Evaluate

Compose application graph
from initial data (e.g. initial
ports) or incomplete graph


First implemented for XCAT
framework


Easy migration to MOCCA


Modification of code
required (xcat.Port)


Similar performance for
XCAT and MOCCA
(exchange of text
documents)

Institute of Computer Science
AGH

18

Communication Intensive
Application Benchmark


Simplified scenario:


2 components


Provides port: receive and send
-
back array of double
(ping
-
pong)


Tested on local Gigabit Ethernet and on transatlantic Internet
between Atlanta and Krakow


2.4 GHz Linux machines


Comparison with XCAT

Institute of Computer Science
AGH

19

Small Data Packets

Factors:


SOAP header overhead in XCAT


Connection pools in RMIX

Institute of Computer Science
AGH

20

Large Data Packets


Encoding (binary vs. base64)


CPU saturation on Gigabit LAN (serialization)


Variance caused by Java garbage collection

Institute of Computer Science
AGH

21

Example: modeling gold clusters


Clusters of atoms


V
ery interesting forms
between isolated atoms or
molecules and solid state


I
mportant for the technology
of constructing nanoscale
devices.


Modeling of clusters


S
everal energy minimization
methods such as

MDSA or L
-
BFGS,


C
hoosing an empirical
potential


H
ighly compute
-
intensive


T
he optimal result depends on
the number of possible
iterations and initial
configurations for each
simulation run.



Institute of Computer Science
AGH

22

Example


deployment

Institute of Computer Science
AGH

23

Integration with existing Grid
middleware



A pool of computing resources may be created by
submitting a number of H2O kernels on many Grid sites


Application components may be deployed on the kernels
belonging to the pool

Standalone
machine
Cluster
Grid node
Resource
Broker
SSH
PBS
LCG
H2O
H2O
H2O
H2O
H2O
H2O
User's
virtual
resource
pool
NS
bind()
lookup()
Institute of Computer Science
AGH

24

Multilanguage support: m
otivation


Grids are heterogeneous


Multiple programming languages


in single application


Java for middleware


C for system programming


FORTRAN for computing


Python for scripting


Multiple protocols


in single application


High speed local networks (Myrinet)


TCP/SSL/TLS in WAN


SOAP for loosely coupled message exchange


Overlay P2P networks for traversing private network
boundaries (NATs)


Context: MOCCA component framework

Institute of Computer Science
AGH

25

Multilanguage Solution
-

Babel


SIDL


Scientific Interface Definition Language


Standard for CCA Components


Supports arrays and complex types


Focus on interfaces


Babel:


SIDL parser


Code generator


Runtime library


Intermediate Object

Representation (IOR)


Core of
B
abel object


Array of function

pointers


Generated code in C

C
C++
f77
f90
Python
Java
Babel
C
C++
f77
f90
Python
Java
Babel
package example version 1.2 {


class Hello {


string hello( in string hello);


}

}


// user defined non
-
static methods:


/**


* Method: hello[]


*/


public java.lang.String hello_Impl (


/*in*/

java.lang.String hello )


{


// DO
-
NOT
-
DELETE splicer.begin(example.Hello.hello)


// Insert
-
Code
-
Here {example.Hello.hello} (hello)


return ”Server says: ” + hello;


// DO
-
NOT
-
DELETE splicer.end(example.Hello.hello)


}

/**


* Method: hello[]


*/

char*

example_Hello_hello(


/*in*/
example_Hello self,


/*in*/
const char* hello);

Institute of Computer Science
AGH

26

Currently: Babel for Local
Applications


All Babel objects in
one process


Implemented in
CCAFFEINE
framework


Existing
multilanguage CCA
components


see
CCA tutorial

Java

application

Fortran

native

library

SIDL

C++

native

library

SIDL

Babel IOR

Babel IOR

Institute of Computer Science
AGH

27

Our Solution


Babel + RMIX


Implementation of
Babel RMI
extensions


generic mechanism
of method invocation
(reflection)


Dynamic loading of
communication
library


No need for code
generation and
compilation

Java

application

Fortran

native

library

SIDL

C++

native

library

SIDL

Babel IOR

RMIX

library

Babel IOR

Network

SIDL

RMIX

library

SIDL

Institute of Computer Science
AGH

28

Beyond CCA ?


Supporting multiple component standards


Goal: to enable loading of components written for different
standards (e.g. Corba CCM, other)


Examples of similar solutions: CCAFFEINE supports „classic”
and „Babel” components; SCIRun2 implementing meta
-
component model


Using MOCCA as promising platform for feasibility studies in
various aspects of Grid components


For experiments with advanced features


Scheduling and load
-
balancing


Fault
-
tolerance


Semantic description and composition


As a platform for higher
-
level grid services and tools

Institute of Computer Science
AGH

29

Fractal/GCM


CCA
Interoperability?


CCA as Fractal


Adapter calls setServices() on a
component


Component registers ports on the
adapter


Component is ready for introspection
and connection


Fractal as CCA


CCA framework creates component,
adapter and invokes setServices()


Adapter introspects the component
and registers interfaces to the
framework


Adapter obtains references to
external interfaces (getPort()) and
binds them to the component


Adapter
CCA
Component
C
C
C
BC
Services
Adapter
C
BC
Services
Fractal
Component
Institute of Computer Science
AGH

30

References


Maciej Malawski, Dawid Kurzyniec, and Vaidy Sunderam.
MOCCA


towards a distributed CCA framework for
metacomputing
, Accepted for: 10th International Workshop on
High
-
Level Parallel Programming Models and Supportive
Environments (HIPS2005) at IPDPS’2005
http://mathcs.emory.edu/dcl/h2o/papers/h2o_hips05.pdf



H2O Project homepage:
http://www.mathcs.emory.edu/dcl/h2o/


CCA Forum:
http://www.cca
-
forum.org


CCA Specification


Tutorial


MOCCA homepage:
http://www.icsr.agh.edu.pl/mambo/mocca


Download binary and source distribution


README