Design and Execution of Scientific Workflows using Web Services

abnormalobeisanceSecurity

Nov 3, 2013 (3 years and 5 months ago)

89 views

Building a KEPLER Extension using
Ptolemy II

KEPLER Collaboration Staff


Presented by: Ilkay Altintas

and
Efrat Jeager

@ San Diego Supercomputer Center,
UCSD


2/20/2004


Building a KEPLER Extension Using Ptolemy II

2

Outline


Overview of Ptolemy II and Kepler collaboration


Ptolemy II


Ptolemy II Architecture


Download, Make and Install


The Vergil user interface and application


Directors and Actors


Ports and Parameters


Manager


Kepler


Overview of pre
-
built Kepler actors and demos


Ingredients to build a Kepler extension


Building you first actor


HelloWorld revisited…



2/20/2004


Building a KEPLER Extension Using Ptolemy II

3

The KEPLER System

for Scientific Workflows …


A framework for design, execution and
deployment of scientific workflows




Caters specifically to the domain scientist



Builds on Ptolemy II


(next slide... :
-
)



Application pull from


various projects

http://kepler.ecoinformatics.org


2/20/2004


Building a KEPLER Extension Using Ptolemy II

4

… based on Ptolemy II


A set of Java packages for heterogeneous,
concurrent modeling, design and execution.


Strengths include:


Precisely defined
models of computation

and
component interaction


e.g. Process Networks (PN)


data
-
flow oriented


An intuitive GUI that lets rapid workflow composition


A modular, reusable and extendable object
-
oriented
environment


An XML based workflow definition


MoML


Workflows defined in Ptolemy II MoML XML schema


Easily exchangeable

2/20/2004


Building a KEPLER Extension Using Ptolemy II

5

Actor
-
Oriented Design

Reference:

PtolemyII

Design Document
-
1, Chapter 1

2/20/2004


Building a KEPLER Extension Using Ptolemy II

6

Focus on Actor
-
Oriented Design

Adapted from the *.ppt slides by

Edward A. Lee (See References)


Object orientation:

class name

data

methods

call

return

What flows through
an object is
sequential control


Actor orientation:

actor name

data (state)

ports

Input data

parameters


Output data

What flows through
an object is
streams of data

2/20/2004


Building a KEPLER Extension Using Ptolemy II

7

Layered Software Architecture

Ptolemy II packages
have carefully
constructed
dependencies and
interfaces

Kernel

Data

Actor

Math

Graph

Adapted from the *.ppt slides by

Edward A. Lee (See References)

2/20/2004


Building a KEPLER Extension Using Ptolemy II

8

Ptolemy II Architecture


Core packages
(actor, data, kernel, math, util)


The data model(
--
abstract syntax) of the models


Abstract semantics


User Interface (UI) packages


MoML and visual interface support


Library packages


Domain polymorphic actors


Domain packages


Implementations of different models of computation

2/20/2004


Building a KEPLER Extension Using Ptolemy II

9

Overview of the Key Classes

2/20/2004


Building a KEPLER Extension Using Ptolemy II

10

Installing Ptolemy II

http://ptolemy.eecs.berkeley.edu/ptolemyII/


User version


Version
Ptolemy II 3.0.2



Web Start Installer


Source download


Precompiled (…downloaded before…)


Source
-
only code


4 steps to make


cd “PTII”


rm

f config.*


./configure


make fast install

2/20/2004


Building a KEPLER Extension Using Ptolemy II

11

Abstract Syntax of PTII Models

Port
Port
Actor
Actor
Link
Relation
Actor
Port
connection
connection
connection
Link
Link
Attributes
Attributes
Attributes
Abstract syntax choices:



Hierarchy is tree
structured (like XML).


A
relation

mediates
connections.


Ports can link multiple
relations and relations
can link multiple ports.


Ports mediate
connections across
levels of the hierarchy
(no statecharts
-
style
level
-
crossing links)




Abstract syntax defines the structure of a
model, but says little about what it means.


Hierarchical Entities, Ports, Connections and Attributes

Adapted from the *.ppt slides by

Edward A. Lee (See References)

2/20/2004


Building a KEPLER Extension Using Ptolemy II

12

Models of Computation


Semantic interpretations of the abstract syntax

Different models <=> Different semantics <=> Different execution



Are actors active? passive? reactive?



Are communications timed? synchronized? buffered?

Models of Computation:




continuous
-
time



dataflow



rendezvous



discrete events



synchronous



time
-
driven



publish/subscribe




process {





read();




}

process {





write();




}

channel

port

port

receiver

One Class of Semantic Models: Producer / Consumer

Adapted from the *.ppt slides by

Edward A. Lee (See References)

2/20/2004


Building a KEPLER Extension Using Ptolemy II

13

Director


Governs the execution of a composite entity.


Scheduling, dispatching threads, generate code, etc.


A composite entity is called
opaque
if it doesn’t
have a local director.


An opaque composite entity inherits the director of its
container as its
executive director
.

2/20/2004


Building a KEPLER Extension Using Ptolemy II

14

Vergil User Interface


The graph editor


Actor library


Director library


Utilities


Ports


Parameters


Quick tour

2/20/2004


Building a KEPLER Extension Using Ptolemy II

15

Using Vergil


Executing a pre
-
built model from the graph
editor


Building a model


Choosing actors


Making connections


Choosing a director


Building composite actors


Saving and running the model


Start Exercise
-
0

2/20/2004


Building a KEPLER Extension Using Ptolemy II

16

MoML


Modeling Markup Language


A primary persistent XML file format for
Ptolemy II.



ptolemy
filename.
xml


ptexecute
filename.
xml


vergil
filename
.xml


moml
configuration
.xml
filename
.xml

2/20/2004


Building a KEPLER Extension Using Ptolemy II

17

An Example MoML File

<?xml version="1.0" standalone="no"?>

<!DOCTYPE entity PUBLIC "
-
//UC Berkeley//DTD MoML
1//EN"

"http://ptolemy.eecs.berkeley.edu/xml/dtd/MoML_1.dtd">

<entity name="test"
class="ptolemy.actor.TypedCompositeActor">

<property name="director"

class="ptolemy.domains.sdf.kernel.SDFDirector"/>

<entity name="ramp" class="ptolemy.actor.lib.Ramp"/>

<entity name="plot"
class="ptolemy.actor.lib.gui.SequencePlotter"/>

<relation name="r"
class="ptolemy.actor.TypedIORelation"/>

<link port="ramp.output" relation="r"/>

<link port="plot.input" relation="r"/>

</entity>


SDF domain

Top
-
level entity

Reference:

PtolemyII

Design Document
-
1, Chapter 6

2/20/2004


Building a KEPLER Extension Using Ptolemy II

18

Type System

2/20/2004


Building a KEPLER Extension Using Ptolemy II

19

Building Kepler Extensions


FOCUS: How to build actors?


Exercises are under:


http://www.sdsc.edu/~altintas/KeplerTutorial/KeplerHandsOn.txt



Please complete the preparation for the
programming exercises…


10 minute break…



2/20/2004


Building a KEPLER Extension Using Ptolemy II

20

Ingredients

We have:


Java 1.4.2


Cygwin for Developers (for Windows users)


Ant 1.5 or higher

Downloaded:


Compiled Ptolemy II source code

http://www.sdsc.edu/~altintas/KeplerTutorial/ptII3.0.2.tar.gz



Kepler extensions (a version from CVS)



http://www.sdsc.edu/~altintas/KeplerTutorial/kepler.zip

Completed the preparation for the hands
-
on
programming exercises.


Coding
-
style guidelines are at:


http://users.sdsc.edu/~altintas/codingStyle.pdf


Please use them!


2/20/2004


Building a KEPLER Extension Using Ptolemy II

21

Adding Actors


Domain and/or data polymorphic actors


Use object
-
oriented inheritance to avoid code
duplication


3 base classes: Source, Sink, Transformer


To use the actors in Vergil


Add them to one of the actor libraries


Most libraries are under $PTII/ptolemy/actor/lib


Libraries are XML files


The basic structure of an actor:


See
http://www.sdsc.edu/~altintas/KeplerTutorial/ActorStructure.txt


2/20/2004


Building a KEPLER Extension Using Ptolemy II

22

Actor Interfaces:
Ports
&

Parameters

input ports

output port

p
1

p
2

p
3

parameters:

a
1

=
value

a
2

=
value

input/output

port

port

Example:

2/20/2004


Building a KEPLER Extension Using Ptolemy II

23

Anatomy of an Actor: Ports


Used for “message transport”, can be an
input, an output, or both.


Key class:
IOPort

(Can be connected to other
IOPort instances via IORelations.)


Use
TypedIOPort
in order to benefit from the
type system! (Domain specific:
DEIOPort
)


Receiver and Sender interfaces depending on
the usage of the port.


Public members of the actors!!!

2/20/2004


Building a KEPLER Extension Using Ptolemy II

24

Ports: Introduction to the API

public
TypedIOPort

portName;
//Definition

//Create the port

portName

= new
TypedIOPort
(this, “
portName
”, true, false);

portName.
setMultiport
(true);
//Can support more than one link

int width = portName
.
getWidth
();

//0 or 1 if single port

//Reading and Writing

portName
.
send
(channelNumber, token);

Token token =
portName
.
get
(channelnumber);

//Setting the type of the port

portName
.
setTypeEquals
(BaseType…a type in the type system…);

portName.
setTypeAtLeast
(…must be a port or parameter…);

2/20/2004


Building a KEPLER Extension Using Ptolemy II

25

Anatomy of an Actor: Parameters


Public members of the actors!


Similar API with ports…


public
Parameter

parameterName;
//Definition

//Creation and setting the initial value: 2
-
ways

parameterName

= new
Parameter

(this,
“parameterName
”,





new Token(…token value…));

OR

parameterName

= new
Parameter

(this,
“parameterName
”);

parameterName.
setExpression
(…tokenValue…);

2/20/2004


Building a KEPLER Extension Using Ptolemy II

26

Anatomy of an Actor: Constructors


The major task:


To create and configure ports and parameters


super(container, name);


Carries the NameDuplication and IllegalAction
exceptions from the super class.


The icon for the actor can be set here.

2/20/2004


Building a KEPLER Extension Using Ptolemy II

27

Exercise
-
1


Create an actor with


An output port


A parameter


Introduction to the fire method.


Adding your actors into the actor library


Using your actors in a workflow

Break for 10 minutes when finished…

2/20/2004


Building a KEPLER Extension Using Ptolemy II

28

Execution of an actor

initialize()




Iterations


wrapUp()

prefire


fire()


postfire()

2/20/2004


Building a KEPLER Extension Using Ptolemy II

29

Action Methods


initialize()
:
Initialize the state variables of
an actor.


prefire
():
Returns a boolean which
indicates if the actor wants to fire.


Can also be used to perform an operation that
will happen exactly once per iteration.


fire
():
The main point of execution.


For reading inputs, producing outputs, read
the current parameter values.

2/20/2004


Building a KEPLER Extension Using Ptolemy II

30

Action Methods (cont.)


postfire
():
Has two tasks:


Updating persistent state


Determining whether the execution of an
actor is complete.


wrapUp
():
For displaying final results.

2/20/2004


Building a KEPLER Extension Using Ptolemy II

31

Things to remember when
implementing a fire() method


Use the methods of the Token class for
arithmetic whenever possible (to get data
polymorphism)


If data
-
polymorphism is not necessary, set the
type to a base type then cast the token to that
type.


Cannot assume that there is be data available at
all the input ports (for domain
-
polymorphism)


Do not update the persistent state in fire() (use
postfire())

2/20/2004


Building a KEPLER Extension Using Ptolemy II

32

Implementing Polymorphism


Class: PortParameterFunction




A
PortParameterFunction

object will be returned as a
function of two objects.


Set the type of the output equal to the type of this object.


Type system will compute the type of the
PortParameterFunction

object and use it as the type of the
output when necessary.


We’ll see this in the example 2!


2/20/2004


Building a KEPLER Extension Using Ptolemy II

33

Exercise
-
2


Getting data through an input port.


Understanding the action methods.


Designing a polymorphic actor.


Using inner classes



Break for 10 minutes when finished…

2/20/2004


Building a KEPLER Extension Using Ptolemy II

34

The manager


Controls the overall execution of a model.


Interacts only with the “
top
-
level composite actor



startRun()
-
> run()
-
> execute()


ExecutionListener interface provides the manager
with info on the events generated during execution.


Hand
-
outs...

show how Ptolemy II handles mixing
models of computations hierarchically.


--
Using opaque composite actors!

2/20/2004


Building a KEPLER Extension Using Ptolemy II

35

Exceptions


A uniform mechanism for reporting errors


Base class:
KernelException


Exception chaining re
-
implemented since Java
versions < 1.4 doesn’t support it.


The detail message includes the detail message from the
cause argument.


A protected _setCause() method is implemented, but not
the public initCause() method that JDK1.4 has.


Non
-
severe exceptions:
IllegalActionException,
NameDuplicationException, NameDuplicationException.


Severe
-
exceptions:
KernelRuntimeException,
InvalidStateException, InternalErrorException.

2/20/2004


Building a KEPLER Extension Using Ptolemy II

36

Exercises
-
3&4


Stopping a model using an actor


Utilizing the manager functions


Handling exceptions


Using the postfire method


Creating a customized icon for an actor


Using an existing code/application in
Kepler.

Break for 10 minutes when finished…

2/20/2004


Building a KEPLER Extension Using Ptolemy II

37

Existing Kepler Actors


Actor prototyping tool


Generic Web and Grid Service


RDBMS Connection and Querying


Generic User Interface and Transformation


Biological Service and Data Access


Rock Classification


EML Data Ingestion


GARP Native Species Pipeline (Using JNI to
utilize C++ code)

2/20/2004


Building a KEPLER Extension Using Ptolemy II

38

Actor Prototyping Tool


Allows “place
-
holder” actors to be defined on the fly by non
-
programmers
during workflow creation


Scientists can thereby create workflows without programming knowledge


Workflows created with these actors can be executed once their
functionality is implemented by a programmer

2/20/2004


Building a KEPLER Extension Using Ptolemy II

39

Distributed SWFs in KEPLER


Web and Grid Service plug
-
ins


WSDL, GWSDL


ProxyInit, GlobusGridJob, GridFTP, DataAccessWizard


WS Harvester


Imports all the operations of a specific WS (or of all


the WSs in a UDDI repository) as Kepler actors


WS
-
deployment interface
(…ongoing work…)


XSLT and XQuery transformers to link non
-
fitting
services together

2/20/2004


Building a KEPLER Extension Using Ptolemy II

40

A Generic Web Service Actor


Given a WSDL and
the name of an
operation of a web
service, dynamically
customizes itself to
implement and
execute that method.

Configure
-

select service


operation

2/20/2004


Building a KEPLER Extension Using Ptolemy II

41

Set Parameters and Commit

Set parameters

and commit

2/20/2004


Building a KEPLER Extension Using Ptolemy II

42

WS Actor after Instantiation

2/20/2004


Building a KEPLER Extension Using Ptolemy II

43

Web Service Harvester



Imports the web services in a repository
into the actor library.



Has the capability to search for web
services based on a keyword.

2/20/2004


Building a KEPLER Extension Using Ptolemy II

44

Composing 3
rd
-
Party WSs

Output of previous

web service

User interaction &

Transformations

Input of next

web service

2/20/2004


Building a KEPLER Extension Using Ptolemy II

45

Current Grid Actors

2/20/2004


Building a KEPLER Extension Using Ptolemy II

46

Providing RDBMS Access


Database connection.


Opening a database connection and passing it to
all actors accessing this database.


Database Querying.


A generic actor that queries a database and
provides its result.


DBConnection type and DBConnectionToken.


A new IOPort type and a token to distinguish a
database connection from any general type.


2/20/2004


Building a KEPLER Extension Using Ptolemy II

47

Database Connection


OpenDBConnection actor:


Input: database connection information


Output: A DBConnectionToken, a reference to a
database connection instance, through a DBConnection
output port.


2/20/2004


Building a KEPLER Extension Using Ptolemy II

48

Querying a Database


DatabaseQuery actor:


Input: A query string (SQL) and a database connection
reference.


Parameters: output type


XML, Record or String.


Process: Execute query. Produce results according to
parameters.

2/20/2004


Building a KEPLER Extension Using Ptolemy II

49

Using Bio
-
Services Actors:

PIW


2/20/2004


Building a KEPLER Extension Using Ptolemy II

50

Sub
-
workflow:
Gene Sequence Processing

Data transformations

between the tasks

2/20/2004


Building a KEPLER Extension Using Ptolemy II

51

Sub
-
workflow:
Transfac Detail

2/20/2004


Building a KEPLER Extension Using Ptolemy II

52

Classifying Igneous Rocks

2/20/2004


Building a KEPLER Extension Using Ptolemy II

53

Actors for Modeling The Classifier

Diagrams information and
transitions between them.

Extracted from the mineral
composition and this level’s
diagram coordinates.

SVG to polygons.

Classifier: Locates
the point’s region.

Finer granularity

Displays the point in the
diagram for this level.

2/20/2004


Building a KEPLER Extension Using Ptolemy II

54

EML Data Ingestion Actor


Ingests any data format described by EML metadata


Converts raw data to Ptolemy format


Data can then be operated on with other actors

2/20/2004


Building a KEPLER Extension Using Ptolemy II

55

Using EML ingestion actor

2/20/2004


Building a KEPLER Extension Using Ptolemy II

56

Third
-
Party Local Applications

2/20/2004


Building a KEPLER Extension Using Ptolemy II

57

What did we learn?


Basic concepts of Ptolemy II


Creating actors within the Ptolemy II framework


The existing Kepler extensions

What we didn’t learn:


Extending the type system


Need to define a class for the new type and add it to the type lattice


Extending the user interface


Effigy, Tableaux (Similar to Model
-
View
-
Controller architecture)


Creating our own director


Developing a director is harder!


Need to inherit the Director class, and implement the semantics of how
the actors will behave. (Semantics==Model of computation)

2/20/2004


Building a KEPLER Extension Using Ptolemy II

58

Ptolemy II Design Document

Volume 1:

User
-
Oriented

Volume 2:

Developer
-
Oriented

Volume 3:

Researcher
-
Oriented

Adapted from the *.ppt slides by

Edward A. Lee (See References)

2/20/2004


Building a KEPLER Extension Using Ptolemy II

59

Becoming a Kepler member?


http://kepler.ecoinformatics.org/members.html


Email me:

altintas@sdsc.edu

to get a read
-
only account.


Will upgrade to become a
contributing member
depending in time.

2/20/2004


Building a KEPLER Extension Using Ptolemy II

60

References


Ptolemy II Design Documentation and Source Code


http://ptolemy.eecs.berkeley.edu/papers/03/ptIIDesignIntro/


Kepler website:
http://kepler.ecoinformatics.org


Overview of Ptolemy II (Powerpoint presentation), Edward A. Lee,
October 9, 2003, presented in course EE249
-

Design of Embedded
Systems: Models, Validation and Synthesis.


An Overview of the Ptolemy Project a nd Actor
-
Oriented Design
(powerpoint presentation), Edward A. Lee, February 4, 2004, OMG
Technical Meeting, Anaheim, CA.


Presentations from the Ptolemy website:
http://ptolemy.eecs.berkeley.edu/ptolemyII/



Kepler documentation, javadocs and this tutorial will be made available
at the website soon.