SCI-BUS WP7/JRA1 - Open Grid Forum

educationafflictedΒιοτεχνολογία

4 Οκτ 2013 (πριν από 3 χρόνια και 10 μήνες)

154 εμφανίσεις

SCI
-
BUS is supported by the FP7 Capacities Programme under contract no. RI
-
283481


Science Gateways for improving the
uptake of Grid and Cloud Computing
Infrastructures

Tamas

Kiss, Stephen Winter, Gabor
Terstyanszky
, Thierry
Delaitre
University of Westminster

Wibke

Sudholt
,
CloudBroker

GmbH

Peter
Kacsuk
,
Zoltan

Farkas
,
SZTAKI

Jens
Krüger
, Sandra
Gesing
,
University of
Tübingen




Using Distributed Computing
Infrastructures is not so easy

gsissh

-
p 2222 grid
-
compute.oesc.ox.ac.uk

globus
-
job
-
submit grid
-
compute.oesc.ox.ac.uk/
jobmanager
-
pbs


-
q RXXXXX /bin/hostname
-
f

globus
-
job
-
status <
uid
>

globus
-
job
-
get
-
output <
uid
>

globus
-
job
-
clean <
uid
>


?

!

or is it?


SCI
-
BUS is a 3
-
year EU FP7 project


Started on
1
st

October
2011


Build
around 30
science gateways to various user
communities in Europe


10
academic
and
5 commercial partners from 10
European countries.

SCIentific

Gateway Based
User Support

Main objectives of SCI
-
BUS


Create a
generic
-
purpose gateway
technology


Elaborate an
application
-
specific

gateway
building technology
and customisation methodology


Establish
production gateway services
both for NGIs and
various user communities


Provide seamless
access to major computing, data and
networking DCIs and services
including supercomputers,
clusters, grids and clouds


Create and maintain a
Liferay

portlet

repository
that enables the
quick creation of user specific customised gateways


Provide
gateway development

and application development
support


Develop
business models
to guarantee the sustainability of the
gateway services and to enable the commercial exploitation

Value brought by the project

SCI
-
BUS will serve large
number of user
communities by providing
generic
-
purpose and
customized scientific
gateways for them.

SCI
-
BUS will also create
and provide access to
many different
portlet

and
application repositories to
help user communities in
sharing and running
these applications via the
SCI
-
BUS gateways

Project partners:


International
seismology community


Helio
-
physics community


Swiss systems biology community of the SystemsX.ch project


German
MoSGrid

computational chemistry and bioinformatics community


Biomedical researchers community of the Academic Medical Centre of the
University of Amsterdam


Astrophysics community


PireGrid

SMEs community


Business process modelling community involving a wide range of areas such as
finance, healthcare, government, production, robotics and emergency


Blender rendering community


Citizen web
-
2 community


Public application developer
community


Subcontractors:


6 sub
-
contractors to be selected representing additional user communities


Hugely competitive


over 3 times as much applications


Additional user communities
supported by the project

The SCI
-
BUS user communities

The gateway services will enable unified access to all the major DCI
infrastructures used in Europe:


Local and remote clusters accessed by queuing systems (PBS,
LSF, Condor, SGE)


Local and remote supercomputers


Local and volunteer Desktop Grids (BOINC, XtremWeb,
OurGrid)


Grids (ARC, gLite, Globus, UNICORE)


Academic Public and Private Clouds (OpenNebula,
Eucalyptus, etc.)


Public Commercial Clouds (Amazon EC2, IBM Cloud, etc.)

The SCI
-
BUS architecture

8

The SCI
-
BUS Infrastructure

The baseline technology

P
-
GRADE
Porta
l
s around the globe


N
ational

Grid

port
als

(
UK, Ireland, Croatia, Turkey, Spain,
Belgium, Malaysia
,

Kazakhstan, Switzerland, Australia,
ClGrid
,
etc
.
)


HP
-
SEE/
SEE
-
GRI
D
-
SCI
,
BalticGrid


US Open Science Grid,
TeraGrid


IDGF (
International Desktop Grid Federation
)


SHIWA
Simulation

Platform


Economy
-
Grid, Swiss
BioGrid
, Bio
és


Biomed V
irtu
al
Organisations
,
MathGrid
,
etc.


HP
-
SEE

P
-
GRADE Portal
s
oftware

family

P
-
GRADE Portal

2.4

NGS P
-
GRADE

portal

P
-
GRADE Portal

2.5

P
-
GRADE Portal

2.8

P
-
GRADE Portal

2.9.1

WS
-
PGRADE

Portal

3.1

WS
-
PGRADE

Portal

3.2

GEMLCA

Grid Legacy

Code Arch.

GEMLCA, storage
concepts

Basic

c
oncep
ts

2008

2009

2010

WS
-
PGRADE

Portal

3.
3

P
-
GRADE Portal

2.
10

Open source
since
2008

Open source
since
2011

201
1

WS
-
PGRADE

Portal

3.4

Motivations

of creating

WS
-

PGRADE/
gUSE


To overcome
(most of
)

the limitations of P
-
GRADE portal:


To provide better modularity


to replace any service


To improve scalability


to millions of jobs


To enable advanced dataflow patterns


To interface with wider range of resources


To separate Application Developer view from Application
User view

WS
-
PGRADE (Web Services Parallel Grid Runtime and Developer
Environment)






and

gUSE

(
Grid

User

Support

Environment
) architecture

WS P
-
GRADE in a nutshell


General purpose
,
workflow
-
oriented
portal.

Supports the
development and execution of workflow
-
based applications


Based on
Liferay


Services

supported by the portal:


New functionalities


Web services


DB connectors


Embedded workflows


Job level PS


Conditional jobs


Recursive graph


Multi
-
generator


Multi
-
collector


CROSS product PS


DOT product PS


Basic Services

EGEE grids


(LCG2,GLite)

Globus grids

(GT2,GT4)

Desktop

grids

clusters

Job execution

File storage

Certificate management

Information system

Brokering

Job monitoring

Workflow &

job visualization

































Inside of the
S
ystem

gU
SE

tires

WF

Graph


editor

WEB
-
UI

(HTML)

Liferay


WS
-
PGRADE
portal

Information

System

WF Storage

File Storage

Application

Repository


WF

Interpreter


Backend

local
submitter

GT4
submitter

Glite

submitter

LCG
submitter

GT2
submitter

GEMLCA
submitter

WS (Axis)
submitter

Glite
Grid

Glite

Grid

GMLCA
Grid

GMLCA
Grid

GT2 Grid

GT2 Grid

GT4 Grid

GT4 Grid

LCG
Grid

LCG
Grid

WS

WS

DCI
-
Bridge

WS P
-
GRADE workflow concept

Concrete Workflow

Algorithms,

Resource
references,

Inputs

Graph

Jobs,

Edges,

Ports

Template

Constraints,

Comments,

Form Generators

Workflow Instance

Running state,

Outputs


Repository Item

Application OR

Project OR,

Workflow part

(G,T,CW)


Legend:

a

b

a
must reference
b

a

b

a

may reference
b

ASM Application Specific
Module


A new module called “Application Specific Module” has been developed
for
gUSE

t
hat

ease the creation of
Application

Specified

Por
tlets



The module

provides programming

API
to control applications developed
using
gUSE


The module

provides programming

API
to access inner
gUSE

services



Any kind of Graphical interface can be developed on top of this module


GUI written in
JSP, JSTL (
Java
Servlet

Tag Library
) and J
ava


GUI can be customized for the grid application


GUI can be customized for end users



ASM
-

Application Specific
Module

DCI systems


Grid

Application

Developer

creates

own

Grid

Application

via

ws
-
PGrade

and export
it

to

the

local
repository


gUSE

Repository


ws
-
PGrade


Grid

Application

Submitter


Workflow
Interpreter


Workflow
Storage


Storage


Information


User space

Grid

Application

Then

the

Grid

Portal

Developer

creates

a
new

web
-
application

(
mainly

an
web
-
interface
)
which

fits

to

the

user

community's

needs
, and
connects

it

to

the

gUSE

using

ASM API

Developed web
-
interface


ASM

Finally

the

end
-
users

create

own

application

from

the

exported

one
,
parameterize

and
execute

it

using

the

developed

portlet

<< import >>

Grid

Application

CloudBroker

Platform


Easy, scalable, secure, integrable and pay
-
per
-
use access to
scientific and technical applications in the cloud


High performance computing application store and marketplace
with accounting and billing


Deployment and execution of applications in the cloud with the
same parameters and files as for local execution


Accessible through the internet via a web browser UI


Integration possibilities into third party tools via web service
-
based
APIs


Public version available under
https://platform.cloudbroker.com



Using infrastructure as a service (IaaS) from cloud providers


Offering platform as a service (PaaS) for software vendors


Providing software as a service (SaaS) to end users

CloudBroker

GmbH

Platform Architecture

CloudBroker

GmbH




CloudBroker Platform

Amazon

Cloud

IBM

Cloud



Cloud

Chemistry

Appli
-
cations

Biology

Appli
-
cations

Medicine

Appli
-
cations

Web

Browser

UI

Engineering
Appli
-

cations

Web Service API

Generic

Workbenches

CloudBroker Integration

Domain
-
Specific
Gateways

R&D End Users and Software Vendors

CLI



Appli
-
cations

CloudBroker

Business Model


Free access to the platform
after registration


Each organization manages
its own users


Prepayment or monthly
billing


Platform usage costs =

resource usage costs +

software usage costs +
CloudBroker usage
surcharges


CloudBroker GmbH

Cloud

Broker

End

Users

Software

Vendors

Cloud

Providers

Resources

Applications

Usage

$

$

$

Clouds and Applications


Currently supported
cloud infrastructures:


Amazon Web Services


IBM
SmartCloud

Enterprise


Adapters to further
clouds planned


Open source
-
based
private cloud tools will be
interfaced within SCI
-
BUS


Current applications:


Chemistry (GAMESS,
Gromacs
)


Biology (BLAST, X! Tandem,
Rosetta)


Medicine (
AutoDock
)


Engineering (
OpenFOAM
)


Further applications in
preparation


Own applications can be
easily added by users

CloudBroker GmbH

CloudBroker

Platform Integration in
Liferay
/WS
-
PGRADE/
gUSE

Gamess

Gromacs

Autodock

Open Foam

Blast

Rosetta

Xtandem

...

API

gUSE


WS
-
PG
RADE

Job
Configuration

Portlet

C
loudBroker

UI
Portlet

UI

DCI
-
BRIDGE


SaaS

(
S
oftwares

as

a Service)

CloudBroker

p
lugin

Expose

the

CloudBroker

UI
through

WS
-
PGRADE
interface

by

proxy.

User

Mapping

Portlet

CB Java
API


Integrate

the

SaaS

capabilities

of CB
into

the

gUSE
/WS
-
PGRADE
system
.


Enable

WS
-
PGRADE
users

to

run

softwares

(
SaaS
)
exposed

by

the

CB platform.


MosGrid

Science gateway for
molecular simulations


Requirement:

Workflow enabled
grid portal for UNICORE based on
Liferay


Solution:


Adaptation of WS
-
PGRADE from
GridSphere

to
Liferay


Development of UNICORE
Submitter


Support for SAML


Development of ASM
(Application Specific Module)
support

MosGrid

Science gateway
-

architecture

MosGrid

Science gateway
-

architecture

MosGrid

Science gateway


application domains


Quantum Chemical Calculations


based on approximated solutions of the Schrödinger equation


Average scalability


Gaussian,
NWChem
,
Turbomole
, …


Molecular Dynamics


based on
forcefields

describing molecular interactions


Good scalability


Gromacs
,
NWChem
, Amber, …


Docking


Based on simplified
forcefields


Excellent scalability


CADDSuite
,
FlexX
, …

MosGrid

Science gateway


application
domains


molecular dynamics

SCI
-
BUS is supported by the FP7 Capacities Programme under contract no. RI
-
283481


Thank you for your attention!

Any questions?

http://www.sci
-
bus.eu/