Running Workflows on Clouds and Grids

feelingmomInternet and Web Development

Dec 7, 2013 (3 years and 11 months ago)

92 views

Running Workflows

on Clouds and Grids

Gabor Terstyanszky, University of Westminster


T. Fahringer, P. Kacsuk, J. Montagnat, I. Taylor

e
-
Science Workshop,

Oxford, 15 March 2012

SHIWA is supported by the FP7 Capacities Programme under contract No. RI
-
261585


2

Workflow Interoperability Challenge

3

3

Computer and Automation Research Institute, Hungarian
Academy of Sciences

MTA SZTAKI

Hungary

University of Innsbruck

UIBK

Austria

Charité
-

Universitätsmedizin Berlin

C
-
UB

Germany

French National Centre for Scientific Research

CNRS

France

University of Westminster

UOW

United Kingdom

Cardiff University

CU

United Kingdom

Academic Medical Centre of the University of Amsterdam

AMC

Netherlands

University of Southern California

USC

USA

SHIWA Project

Partners
:

Duration
:

July

2010



June

2012

(extended

to

September

2012
)

Project Objectives and Services

Objectives
:


To

create

an

environment

which

enables

seamless

execution

of

workflows

of

different

workflow

systems

through

workflow

interoperability

Services
:


To

support

the

whole

workflow

lifecycle
:

editing,

uploading,

browsing

downloading

and

executing

workflows



To

provide

coarse
-

and

fine
-
grained

workflow

interoperability

solutions



To

offer

Distributed

Computing

Infrastructure

interoperability

solution


To

provide

desktop

computer

and

portal

interfaces

to

manage

workflows

Key

actors
:


researchers



workflow

engine

developers






workflow

developers

4

5

Workflow description
abstract workflow


plain text which describes the inputs and outputs and explains what the workflow does

Workflow implementations
concrete workflow


It contains all implementation files or references to these files (via
eg
. URLs) and also holds other
data/metadata necessary to run the workflow on its associated workflow engine.

Workflow configurations


It contains pre
-
defined input files and parameters or references to these files and parameters and other
data/metadata of a workflow.

Workflow engines


It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references to
them

SHIWA Data Structure

6

J2

WF4

J1

J3

GEMLCA
client

GEMLCA
Service

Workflow
Engine B

Workflow of Workflow
Engine B

Workflows of
Workflow Engine A

Coarse
-
Grained Interoperability

DCI


native

jobs

(or

workflows)
:

J
1
,

J
2
,

J
3


non
-
native

workflows
:

WF
4


-

black

boxes

which

are

managed

as

legacy

code

applications


Workflow
Engine A

7

SHIWA Science Gateway

Resources

native WE

WS
-
PGRADE

local resources
: invocation of locally deployed WEs
portal

WS
-
PGRADE v3.3.2


WE submission to local cluster

repository

GEMLCA + SHIWA repo

remote resources
: through remotely pre
-
deployed


submitter

GEMLCA with GIB

WEs to
gLite

and
Globus

DCIs

proxy management

SHIWA Proxy Server



SHIWA Portal

Legacy
Code GUI

SHIWA Science Gateway

WE1

WEp

GEMLCA Repository

WF1

WFm

SHIWA Repository

WF1

WFn

WS
-
PGRADE
Workflow
engine

WS
-
PGRADE
Workflow
editor

gLite DCI

Globus

DCI

pre
-
deployed
-
WEs

MOTEUR
WE

Kepler
WE

Taverna
WE

Triana WE

local
cluster

MOTEUR
WE

GWES WE

ASKALON
WE

SHIWA VO

ASKALON
WE

GEMLCA Service

GEMLCA
with GIB

SHIWA Proxy Server

Proxy
Server

CGI Infrastructure

ProActive
WE

8

SHIWA Portal: Editing Workflows

9

SHIWA Portal: Configuring Workflows

10

SHIWA Portal: Executing Workflows

11

SHIWA Workflow Repository

Browse
Workflows

Browse
Implementations

Search

Sort

12

SHIWA Workflow Repository

13

SHIWA Repository

SHIWA Portal

WF1

SHIWA Science Gateway

GEMLCA Service

WFn

WE1

WEp

GEMLCA Repository

WE + WF

WF1

WFm

GEMLCA
with GIB

WF list

WS
-
PGRADE
Workflow
engine

WS
-
PGRADE
Workflow
editor

edit WF

s2

search WF

s1

s5

s4

gLite DCI

MOTEUR
WE

GWES WE

Globus

DCI

pre
-
deployed
-
WEs

MOTEUR
WE

Kepler
WE

Taverna
WE

Triana WE

local
cluster

ASKALON
WE

SHIWA VO

ASKALON
WE

user

invoke WE

s7

CGI User Scenario: Native WE

Legacy
Code GUI

SHIWA Proxy Server

Proxy
Server

s3

s7

submit WE

s6

ProActive
WE

14

14

14

SHIWA Repository

SHIWA Portal

WF1

SHIWA Science Gateway

GEMLCA Service

WFn

WE1

WEp

GEMLCA Repository

WF1

WFm

GEMLCA
with GIB

WS
-
PGRADE
Workflow
Engine

WS
-
PGRADE
Workflow
Editor

search WF

s1

WE + WF

s5

gLite DCI

MOTEUR
WE

GWES WE

Globus

DCI

MOTEUR
WE

Kepler
WE

Taverna
WE

Triana WE

local
cluster

ASKALON
WE

SHIWA VO

ASKALON
WE

user

invoke WE

s7

CGI User Scenario: Non
-
native WE

Legacy
Code GUI

GEMLCA
Client

MOTEUR
Workflow
Engine

MOTEUR
Workflow
editor

s2

s3

GEMLCA
UI

SHIWA Proxy Server

Proxy
Server

s6

s4

ProActive
WE

15

DCI Interoperability

JSDL
Translator

Workflow
Engine

DCI
Bridge

Metabroker

Proxy
Server

DCI

workflow for DCI B

J2

J1

J4

J3

jobs in JSDL

J2

J1

J4

J3

jobs in non
-
JSDL

16

DCI Bridge

Features
:


w
eb

service

based

app

which

provides

standard

access

to

different

DCIs
:


clouds

-

EC
2
,

OpenNebula


clusters

-

LSF,

PSB



d
esktop

grids

-

BOINC,

OurGrid
,

XtremWeb


service

grids

-

ARC,

gLite
,

Globus
,

Unicore


workflow

jobs

are

submitted

to

the

BES

Factory

Service

of

the

DCI

Bridge

where

jobs

are

described

in

JSDL

Components
:


resource

registry

-

configuration

interface

of

DCIs


a
pplication

manager

-

BES

management


runtime

system

-

job

execution

management

via

BES

Factory


m
onitoring

-

logs

and

messages

handling

and

visualisation

17

DCI Bridge

WS
-
PGRADE
Workflow
Engine

WS
-
PGRADE
Workflow
Editor

Portal

BES Factory
Service

Job

Registry

Input

Queue

Upload
Marnager

Plug
-
in
Manager

Plug

inA

Plug

inB

Plug

inC

inputs & executables

outputs

clouds

desktop
grids

service

grids

DCI Bridge

proxy server

s1

s2

s3

s4

s5

s6

s7

s8

s10

s9

s11

18

18

18

Fine
-
Grained Interoperability


front
-
end

plug
-
in
:

converts

workflow

of

workflow

engine

A

into

IWIR

(Intermediate

Workflow

Interpreter

Representation)


back
-
end

plug
-
in
:

converts

from

IWIR

into

workflow

of

workflow

engine

B


IWIR workflow

J2

J1

J4

J3

WFA
-
>IWIR
frontend
plug
-
in

J2

J1

J4

Workflow of Workflow Engine A

J3

WFA
-
>IWIR
backend
plug
-
in

Workflow
Engine B

DCI

J2

J1

J4

J3

Workflow of Workflow Engine B

ASKALON

MOTEUR

PGRADE

Triana

SHIWA Repository

SHIWA Portal

WF1

SHIWA Science Gateway

WFn

WE1

WFp

GEMLCA Repository

WF1

WFm

gLite DCI

Globus DCI

DCIs

ARC DCI

Unicore
DCI

GEMLCA
submitter

Kepler WE

GWES
WE

Taverna
WE

Triana WE

MOTEUR
WE

ASKALON
WE

WS
-
PGRADE
Workflow
engine

WS
-
PGRADE
Workflow
editor

PGRADE
WE

pre
-
deployed
-
WEs


SSP DCI & FGI Infrastructure

SHIWA Science Gateway

Resources

native WE

ASKALON, MOTEUR, WS
-
PGRADE

local WEs
:
Kepler
, MOTEUR ,
Taverna

&
Triana

WEs

portal

WS
-
PGRADE v3.4.1


submit to the local cluster

repository

GEMLCA + SHIWA

remote WEs
: ASKALON, GWES & MOTEUR WEs

GEMLCA





ARC,
gLite
,
Globus

and
Unicore

DCIs

proxy server






DCI
Bridge

Meta
Broker

Proxy
Server

JSDL
Translator

ProActive
WE

20

20

SHIWA Repository

SHIWA Portal

WF1

SHIWA Science Gateway

WFn

WE1

WFp

GEMLCA Repository

WF1

WFm

gLite DCI

Globus DCI

DCIs

ARC DCI

Unicore
DCI

WS
-
PGRADE
Workflow
engine


DCI Interoperability Scenario

DCI
Bridge

Meta
Broker

Proxy
Server

JSDL
Translator

search WF

s1

s6

s4

s2

edit workflow

s7

submit
workflow

s5

s3

s3

WS
-
PGRADE
Workflow
editor

21

21

SHIWA Repository

SHIWA Portal

WF1

SHIWA Science Gateway

WFn

WE1

WFp

GEMLCA Repository

WF1

WFm

DCIs

WS
-
PGRADE
Workflow
Engine

WS
-
PGRADE
Workflow
Editor


FGI Interoperability Scenario

c

DCI
Bridge

Meta
Broker

Proxy
Server

JSDL
Translator

search WF

s1

Workflow
Engine

Workflow
Editor

s4

FP1

FP2

FPn

c

BP1

BP2

BPm

s3

s2

edit WF

submit WF

s7

gLite DCI

Globus DCI

ARC DCI

Unicore
DCI

s5

s6

22

22


SHIWA Simulation Platform:

Access





S
HIWA

Science

Gateway

(SHIWA

Portal

+

SHIWA

Repository)


joint

portal

and

repository

account

given

by

the

science

gateway

administrator


DCI

resources


certificate

given

by

the

relevant

DCI

administrator


Access

to

the

SHIWA

Portal


shiwa
-
portal
.
cpc
.
wmin
.
ac
.
uk/liferay
-
portal
-
6
.
05


Access

to

the

SHIWA

Repository


shiwa
-
repo
.
cpc
.
wmin
.
ac
.
uk





23


Conclusions


the

simulation

platform

supports

the

whole

lifecycle

of

workflows
,

i
.
e
.

creating,

testing,

uploading,

browsing,

downloading

and

running

workflows



users

can

use

workflows

of

their

own

and

of

other

workflow

systems

through

the

same

user

interface,

i
.
e
.

they

can

re
-
cycle

or

share

workflows

created

by

other

research

teams

or

even

other

research

communities



workflow

developers

(or

e
-
scientists)

can

create,

run,

upload

workflows

through

the

simulation

platform



researchers

(astro

physicists,

earth

scientists,

bio

scientists,

etc
.
)

can

browse

the

workflow

repository,

select

and

run

workflows

through

the

simulation

platform



the

simulation

platform

provides

access

to

major

European

e
-
infrastructure
,

the

European

Grid

Initiative

(EGI)

infrastructure

which

offers

large

pool

of

compute

and

storage

resources