Building Computational Communities from Federated ...

jetmorebrisketSoftware and s/w Development

Aug 15, 2012 (4 years and 10 months ago)

204 views

RealityGrid: An Integrated Approach to Middleware
through ICENI


Prof John Darlington


London e
-
Science Centre,

Imperial College London, UK




Positioning


Grids


transparent
mapping of complex
applications onto distributed

machinery





Routinely


for practising scientists c.f.
heroic HPC

Requirements



Simple application construction


Automatic mapping to appropriate machines


Automatic scheduling of activities


Simple support for user interaction


Simple support for collaboration



Straightforward middleware installation and
maintenance





No half
-
way house


need complete solution



Requires automated use of considerable
knowledge/intelligence previously provided manually



Requires complete set of interoperable services for
whole task



Not necessarily monolithic single middleware
solution (c.f. heroic middleware)

For Reality Grid



Efficient execution of LB3D code in Grid
environment



Integrated support for collaborative steering
and visualisation


The application Pipeline


Deployment


Getting code and data to the resource


Execution


Running the code


Steering


Results Analysis


Visualisation


Either after completion or real time

ICENI: Imperial College e
-
Science
Network Infrastructure


Integrated Grid Middleware Solution


Interoperability between architectures, APIs


Added value layer to other middleware


Usability: Interactive Grid Workflows


Deployment: Complete Install from Webstart


Role and policy driven security


Foundation for higher
-
level Services and
Autonomous Composition


ICENI Open Source licence (extended SISSL)

http://www.lesc.ic.ac.uk/iceni/

Deep Track:

Tackle fundamental issues within Grids


Focus on aspects relevant to RG scientists


Running jobs & selecting resources


Staging and managing data


Collaborative steering and visualisation


Controlled sharing of resources, data & knowledge


Export solutions to other Grid activities


Promote best practice through RG experience


Lead & develop relevant grid standards

Activity over the last year…


Use ICENI to launch LB3D


Select which LB3D instance to use


Consider machine availability & basic performance


Wrapping LB3D as a binary component


Visualise & steer LB3D through ICENI


Integration of fast track file
-
based steering


ICENI testbed


Simplify deployment through Webstart


Wizards to simplify configuration

Resource Discovery & Initiation

Select any machine with LB3D

Select a specific machine with LB3D

Dynamic Discovery & Composition

Register as running
component services
in the NetBeans user
interface

Deployed application

Add new advertised
components

Drag
-
and
-
drop
running component

Execute to create new component
instances and connect to
application

Application

Visualisation

Server

Collaborative Visualisation & Steering
Integrated with ICENI Driven Access Grid!

Visualisation
server

Application
component

Rendering
engine 1

Rendering
engine 2

Streamed to Access Grid

Visualisation client 1

Visualisation client

2

Service Oriented Architecture

Dataset A & B

Dataset A

Dataset B

View of
dataset A

View of
dataset B

Installation Mechanism and Control Centre

Client Requirements:



JRE 1.4.2



Java Web Start (inc.)



Internet Access

Centralised configuration and service execution

The ICENI Control Centre now has an installation ‘wizard’ that
encapsulates configuration & execution for standard actions.

Wrapping up Legacy Code

Legacy code can quickly be made

available to the ICENI architecture

Story so far… within Grids


Can submit & run jobs


But don’t necessarily know when they will run


Collaborative visualisation & steering


Need to co
-
ordinate multiple resources


Require predictability & guaranteed execution

Effective Interaction within the Grid

Application

Fabric

Execution

Performance model
to predict resource
requirements

Obtain resource
reservations

Execution monitoring
to build performance
database

Service Architecture

Scheduler

Reservation

Service

Performance

Store

Launcher

Application

Service

Reservation

Engine

Performance Events


Events fired whenever ICENI components
start or ports are accessed


Used to gather performance information
about currently running application


Events contain data relating to:


Time & application


Source component type, location & resource


Event type: start or port


Events are serialised objects


Can be XML documents

Collection of Performance Results

Data Collector

Linear Equation

Source

Linear Equation

Solver

Display Vector

Results

Time Event


12:00 Linear Equation Source Start


12:04 Send out Equations


12:03 Linear Equation Solver Start


12:05 Receive Equations


12:12 ………..

Event:

Start Linear

Equation Source

Performance Processing

Workflow

Performance Store


Fully exploit meta
-
data to infer
temporal view of workflow

Length:

Execution

Time

Width:

Resource

Usage

From the data flow and performance database infer the temporal workflow

and thereby which component must be executed where and when.

Reservations need to obtained from the grid fabric.

Launching Service


Have a generic job submission system


This is being developed into an independent
Web Service (GridSAM)

GridSAM

GridSAM

OMII Distribution

SGE

GridSAM

Condor
-
G

Tomcat & Axis

GridSAM

Shell

Condor

Application/

User

Evolution of ICENI

Future work…


Expansion of ICENI testbed with performance
driven scheduling & reservations


Use of simple WS to start jobs


Prototype in advanced development


Integration of service based steering

BINARY COMPONENT

ICENI

Steering

BASIC

APPLICATION

FILE STEERED

APPLICATION

GS STEERED

APPLICATION

Acknowledgements


Director: Professor John Darlington


Research Staff:


Nathalie Furmento, Stephen McGough, William Lee


Jeremy Cohen, Marko Krznaric, Murtaza Gulamali


Asif Saleem, Laurie Young, Jeffrey Hau


David McBride, Ali Afzal


Support Staff:


Oliver Jevons, Sue Brookes, Glynn Cunin, Keith Sephton


Alumni:


Steven Newhouse, Yong Xie, Gary Kong


James Stanton, Anthony Mayer, Angela O’Brien


Contact:


http://www.lesc.ic.ac.uk/


e
-
mail: lesc@ic.ac.uk