Download

quaggahooliganInternet and Web Development

Feb 5, 2013 (4 years and 9 months ago)

116 views

GREDIA
-

GRid enabled access

to rich mEDIA content

Dr. Vasilios Tountopoulos

Presentation Outline


GREDIA Motivation


Objectives and Research Challenges


Innovation and Advances


Scientific Achievements


GREDIA Architecture


Assessment to real life use cases

Motivation


Significant Advances
in Grid technology


Enhanced Grid
middleware


Grid for data and
computational
expensive tasks



But...

How to exploit
Grid
technology?

Restrictions
in the use of
Grid
Technologies

Business
application

developers

Grid Experts

How to address
specific
requirements

No
investment
on immature
markets

What is needed?


Make Grid application development
approachable


Simplicity of the application development
process


Completeness of addressing critical
components of the produced applications


Provide solutions for incorporating mobile
devices as terminal handsets with
increased functionalities

Here comes...

Partially funded by the EC under FP6

a

generic

platform

that

will

combine

new

and

existing

Grid

middleware

in

order

to

facilitate

the

provision

of

business

services

which

mainly

demand

access

and

sharing

of

large

quantities

of

distributed

content

a generic platform

both

desktop

and

mobile

terminals

are

addressed,

with

advances

proposed

relating

to

extensions

that

will

facilitate

mobile

devices

to

exploit

Grid

technologies

in

a

seamless

way

with mobile access

the

potential

effects

of

the

platform

will

be

validated

through

two

pilot

applications

servicing

news

and

banking

organisations,

demonstrating

mobile

access

to

distributed

multimedia

content

and

computationally

expensive

decision

support

for

critical

banking

operations

relying

on

distributed

complex

data

demonstrated with two pilot applications

The concept

Develop

a

reliable

platform

with

high
-
level

support

for

the

design,

implementation

and

operational

deployment

of

secure

Grid

business

applications

a
Grid
application development platform

The

platform

will

make

use

of

existing

and

new

middleware

which

will

be

built

to

service

business

applications

in

need

of

secure

access

to

distributed

rich

media

content

generic

Grid middleware

The

platform

will

support

services

that

will

be

built

to

allow

mobile

devices

to

participate

in

the

Grid

Virtual

Organisation

mobile services

The

platform

data

and

transaction

will

be

protected

at

all

levels

through

a

dedicated

security

framework

that

will

be

developed

for

these

purposes


a
security framework

The

platform

will

be

validated

by

producing

two

pilot

Grid

applications

to

service

the

domains

of

media

and

banking

validate the platform

The objectives

The Advances

Application Development
Platform
Management of
Virtual
Organisations
Mobile Grid
Enhanced Grid
Middleware
Security Framework
Application Development
Platform
Management of
Virtual
Organisations
Mobile Grid
Enhanced Grid
Middleware
Security Framework
Innovative

search

and

retrieval

mechanisms

for

fast

access

of

annotated

numerical

and

multimedia

content

distributed

over

the

Grid,

based

on

P
2
P

overlay

networks

technology

generic Grid middleware

A

new

notation

for

Grid

application

developers

which

will

allow

for

easy

definition

of

Grid

application's

static

(structure)

and

dynamic

(control

flow)

aspects

Grid application development platform

Ontological

services

supporting

interoperability

in

and

across

VOs,

building

a

semantic

abstraction

layer

over

available

resources

in

the

Grid

and

utilising

knowledge

evolution

virtual organisation management

Implementation

of

information

services

that

rapidly

catalogue

content

on

mobile

devices

and

design

of

Agents

that

seek

data

caching

and

replication

services

based

on

QoS

criteria


mobile Grid environments

Address

security

issues

on

the

Grid

as

authentication

of

entities,

confidentiality

and

integrity

to

accesses

from

remote

resources

and

composibility

of

a

wholistic

solution

a security framework

The Research challenges

The Novelties


Massive, distributed and scalable data processing


Proactive runtime service discovery based on different
criteria


Breakthrough and advanced capabilities to mobile devices


Both data and services can be provided through mobile
devices


Flexible and self
-
contained workflow development and
execution environment, enabling application developers
to use a scripting approach towards workflow
development


A framework for flexible rights and contract management


Distributed and across
-
organisational contract
negotiation


Contract
-
based security attributes


Rapid application development on Symbian OS using
Ruby 1.9

Impact on Scientific
Research


Produce a reliable application development
platform for:


Building business applications


Enabling mobile devices to participate in the Grid
Virtual Organisation in a seamless way


Protecting data at all levels of the Grid Virtual
Organisation


Enhance Grid middleware for:


Enabling access to distributed annotated numerical
and multimedia content in a secure way.

Architectural View

Main Achievements

Research Challenge

GREDIA Component

Rich Data Location Service (RDLS)

Generic Grid middleware

Grid Run Time
Service Discovery Tool (GRSD)

Grid application development platform

Application Development Platform (APPEA)

Framework for Intelligent Virtual Organizations
(FiVO)

Virtual Organisation management
and
Security

GREDIA Security framework

Mobile Proxy

Mobile Grid envi
ronments

Ruby 1.9VM


GREDIA Contents


RDLS
-

Rich Data Location Services


GRSD
-

Grid Runtime Service Discovery Tool


APPEA
-

Grid Application Development Platform


FiVO


Framework for intelligent Virtual
Organizations


Security Framework: PERMIS
-
based authorisation
module


Mobile Grid


Proxy


Ruby 1.9VM

RDLS Objectives


Definition of unified metadata schema for rich

multimedia content


MPEG
-
7 standard


Efficient search and location of geographically disperse data
items


Peer
-
to
-
Peer techniques for scalability and fault tolerance


Distributed Replica Location Service (DRLS)


Indexing of annotated content for fast search


Space Filling Curves method


P2P
-
based data transfer mechanism


Modified BitTorrent and GridFTP Protocols


䝲楤呯牲敮G


Provision of core functionalities/services for manipulation of
data


indexing, storage, search, retrieval

Gredia Middleware

Middleware Components


Multidimentional Indexing Component


The metadata XML provided by the user is
indexed and placed in the query enabled
P2P overlay


Data transfer and replication
component


The client retrieves chunks of files from
different locations using the Gridtorrent
protocol


The client dynamically chooses replica
locations to place its data

Multidimensional Indexing


Indexing mechanism must support


Complex queries


indexing according to multiple attributes


Point and Range queries


Space Filling Curves


Division of hypercubic space in sub
-
hypercubes


Recursive traversal of every sub
-
hypercube


Mapping of N
-
dimensional space to 1
-
dimensional space

00
01
10
11
00
01
10
11
Hilbert
Space
-
Filling
Curve
00
01
10
11
00
11
0000
0001
0010
0011
0100
0101
0110
0111
1000
1001
1010
1011
1100
1101
1110
1111
Distributed RLS


Centralized RLS


Local Replica Catalogue


Replica Location Index


Distributed RLS


Kademlia DHT overlay


(key, value) = (LFN, PFN)


Key is the hash of the LFN


Decentralized solution


Variable deployment
modes:


Real life scenario, one
node per server


Simulation, all nodes in
one server


Extended Simulation, N
servers, M nodes/server

Gridtorrent Protocol


Metadata fields


Current Id


File size


Piece size


Hashes


Distributed RLS instead of
Tracker


Partial GridFTP for actual
transfer


BitTorrent replica selection
and tit
-
for
-
tat algorithm.


Compatible with plain
GridFTP servers


PFN’s prefix determines
protocol
(
gtp://site.fully.qualified.doma
in.name/path/to/file)


Bittorrent
Peer
3
GridFTP Server
New connection
New connection
Peer set
Peer
2
GridTorrent client
Peer
1
GridTorrent client
Bittorrent
Striped GridFTP
Striped GridFTP
CurrentID
:
pfn
1

pfn
2

pfn
3

publish

pfns
Peer that wants to
download a file
size
,
piece
_
size
,
hashes
,
currentID
Distributed
RLS
Striped GridFTP
Bittorrent
New connection
piece
piece
piece
piece
piece
piece
Deployment Scenario


Seed registers file in DRLS


GT Client connects to the
other three daemons and
issues a “start download”
command


GT daemons get a list of
the active peers from the
DRLS


GT daemons download
pieces from the seed and
exchange pieces with the
other two


FILE is replicated to the
other 3 servers

GRSD Tool


Execution time identification of
services that become


Unavailable


Malfunctioning


Unused due to context changes


Services are identified based on
synchronous and asynchronous
queries




ASE 2008


© Andrea Zisman


Motivation (Example)






S1: Find journalists

S2: Assign tasks

S3: Combine material

S4: Broadcast on PDA





S1: unavailable

S3: context change


(too slow)

application context

change (no wifi;cost)

new “better” service

S1’: Find journalist, (24/7)




ASE 2008


© Andrea Zisman


GRSD Framework


Service discovery based on
different
characteristics

of the system and services to be
replaced (structural/behavioural/quality/contextual)


Use of both
pull

and
push

modes of query
execution


Proactive
service discovery execution: services are
identified in parallel to system execution


Queries include
structural
,
behavioural
,
quality
, and
contextual aspects


Discovery is executed in a two
-
stage process based
on
similarity analysis

and
distance measure


Service specifications represented as
facets

GRSD Architecture

Service Discovery Query


SerDiQueL:
Ser
vice
Di
scovery
Que
ry
L
anguage



Three Criteria:


Structural: represent interface of a required service


Behavioural: represent functionality of a required
service based on temporal logic constructs


Constraints: represent extra quality, structural, or
behavioural aspects of a required service


Contextual: information that changes dynamically


Non
-
contextual: static information


Hard: it must be satisfied by all discovered services
(filtering)


Soft: it is used to rank candidate services (ranking)

Matching process


Two
-
stage process:


Hard non
-
contextual constraints
(Filtering)


Structural and behavioural criteria


Soft non
-
contextual constraints
(Ranking)


Contextual constraints



Ranking
based on distance measures calculated
between query and services






OD(Q, S) = (D
S
tructural_Behavioural

(Q,S) + D
Soft_non
-
context
(Q,S) +


D
Context
(Q,S)) / |D|


Dynamic Service Discovery


Pull query execution mode:


Identification of services
that are initially bound to
an application and of candidate services


Pre
-
step for push query execution mode


Support for changes in the context of an application


environment


Push query execution mode:


Identification of services that need to replace a
service in a running application


Replacement services are identified in parallel to the


application’s execution (proactive discovery)


Based on subscribed services, queries, and
application environment


Asynchronous events are pushed into listeners


Use of replacement policies for subscribed services

APPEA

Objectives


Platform for building
collaborative Grid applications


Provides uniform and transparent way of accessing
distributed data repositories, resources and services
through scripting language


Enables
Service Developers

to register grid services


E
nables
S
cenario
D
evelopers

to prepare application

scenarios


JRuby, enhanced to manipulate Grid Objects


Enables
Application Users
to use pre
-
developed
scenarios

APPEA Architecture


Preparation of
application scenarios


Use of Grid services


Map between
technical and
abstract description


Use of data sources


Call upon actors
registered with the
VO framework


Execution of
application scenarios


Expressed in scripts


Execution Service


GSEngine


Full
-
featured remotely
accessible
service for
scenario enactment


Accessible through proprietary
protocol over TCP/IP stack
concealed by Java client

provided


Supports scenarios with API
for
:


interacting with scenario actors


accessing distributed files


building user interface


logging, tracing
and much
more


Grid Object Invoker


Uniform access to computational resources and mobile devices within
scenario script

Grid Resource Registry


Objective: To d
evelop a repository
that stores information about all
available and accessible
computational resources, hides the
ir

technological complexity and
provides uniform and user
-
friendly
access to web services, component
and
G
rid infrastructures


Two layers of resource description


technology
-
independent


technology
-
specific


ResBrowser Plugin assists during

application developmen
t


GRRAdmin Plugin allows managing
information stored inside the
registry

Monitoring System
Registry Notification Listener
notify about resource status
Registry Service
GRSDT
Registry Core
Registry DB
<<artifact>>
SQL queries
Get information required by GRSDT to find new service
Set Grid Object Instance
status/properties
Delegate request
Grid Resources Registry
Presentation
Gredia
portal
Appea
Search about resources
informations, register/edit resource
Scheduler
Grid Object Invoker
Find Grid Object Instances
Get Grid Object
Instance properties
Internal components
External components
OR
mapping
Search, add/edit/remove resource
RSM Registry


Objective: to provide functionality of
storing and managing the semantic
information for grid objects and actors


The AEPT RSM
-
R Plugin


Allows user to annotate and browse the resources in
the VO


Assist script writing by automatically generation of
codes for querying resource


The RSM
-
R Service Interface


Enable run
-
time access to the functions of RSM
-
R in
scenario scripts

FiVO


Semantic framework


Definition and Management of Virtual
Organizations (VO)


Participation of users in VOs through signing of
Contracts


Development of business oriented knowledge
model of VO


By means of generic ontologies



Semantic metadata specification


Ontology
-
based discovery modules

FiVO Overview

VO
-
1
inetOrgPerson
inetOrgPerson
VO
-
1
CONTRACT
inetOrgPerson
inetOrgPerson
inetOrgPerson
ORGANIZATION A
ORGANIZATION B
ORGANIZATION C
inetOrgPerson
inetOrgPerson
inetOrgPerson
inetOrgPerson
inetOrgPerson
inetOrgPerson
inetOrgPerson
ORGANIZATION D
(
GREDIA Middleware
Provider
)
APPEA
MDS
MDS
Service
Registry
Service
Registry
Service
Registry
FiVO
FVO
FiVO
FiVO
inetOrgPerson
inetOrgPerson
MDS
inetOrgPerson
inetOrgPerson
RDLS
RDLS
RDLS
Outcome


Contract ontology

FiVO Functionalities


Contract negotiation formal model


Formal contract negotiation model


Distributed Web Service infrastructure for negotiation VO
contract


Ontological GUI for contract negotiation process


Contract model


Contract statemet


logical union of basic logical statements over
organization’s resources


S ={ (p
1
, p
2
, …): n in N and p
i

= (a, x
i
, z
i
, d
i
), i = 1,2, .. ,n}

w
here a


agent (actor), x
i



resource, z
i



action, d
i

-

attributes


Security


Authorization and Access Control Mechanisms


Certificate based authentication


Security Enabled FiVO

Mobile Grid


Provision of core mobile Grid Services


Extension of low level Operating System functionality to the
Grid


Symbian OS through Ruby Virtual Machine API


The mobile user can


connect to the Grid


upload data to the repository


Publish/consume a service to the Grid


through the
Proxy


Mobile


Proxy interaction


Caching mobile data


Synchronization

Mobile Grid Architecture


Mobile device technically is
not

a grid node.


Proxy
is
a grid node with a ‘mobile facing’ part and
a ‘grid facing’ part

Mobile proxy architecture

Grid client
Proxy
Mobile node
Gateway
/
firewall
/
router
redirector
connector
grid services server
Mobile data cache
Mobile data
Internet
Internet
/
WLAN
/
PAN
...
THE
‘STATIC’
GRID
THE
‘’MOBILE’
GRID
replication
/
synchronisation
Internet

Use of mobile devices
on the grid


Provide dynamic
content


Expose services


Run services on the
grid


Gateway/firewall
restrictions

䵯扩汥M
偲潸P


Bridge ‘mobile’ and
‘static’ grid network

Ruby 1.9VM Objectives


Investigate

how can researchers rapidly
develop and prototype their ideas on
mobile phones


Prove

this concept by allowing researchers
to integrate grid services on mobile phones



For this reason Symbian decided to enable
researchers by providing them with the Ruby
1.9.0 programming environment on Symbian
OS devices.



Native Symbian C++ vs Ruby


To write a Symbian C++ application requires:


The relevant SDK for S60 or UIQ


Knowledge of that SDK and Symbian C++


Carbide.c++, Carbide.vs or a text editor with
command line if you
enjoy

typing out the
application framework


A Windows
-
only compiler


To write a Ruby script requires:


Knowledge of Ruby


On
-
line tutorials available


The Ruby virtual machine


A text editor

The Symbian OS Ruby 1.9 VM port


Moved from our pre
-
release Ruby
1.9
port to
Ruby
1.9.0.0
and then to the now public
1.9.1
p
0


Added TCP/IP support


Added camera support


Added LBS recording support


Added battery level and signal strength support


Added OpenSSL support


Added I/O capabilities so that the user can
interact with the VM (IRB for example)



Added a more user friendly program launcher


Ported other minor extensions (zlib, fctl, digest)


Application Scenarios


Media Pilot Application




Banking Pilot Application

“Allow journalists and photographers to make their work available
to a trusted network of peers at the moment it is produced, either
from desktop or mobile devices”

“Enable the exchange of private information between banking
organisations and their potential customers for the assessment

of their creditability and the risk associated with granting a
specific loan according to the Basel II regulations”

Media Pilot Application

Banking Pilot Application

Conformance to EC Initiatives

Potential contribution
to NESSI




Service
-
oriented utility infrastructure


Design, development and operational deployment of
scalable approaches to Service Oriented Infrastructures


Offer for a technology bases for wide scale Cloud
computing utility, leveraging the interoperation between
open source software components




Service and System Engineering


Research on dynamic service
discover


Identification of services to replace
participating services in Grid service
-
based business applications




Support mobility and wireless as the key drivers
towards a “Future Internet” architecture


Enable mobile devices for participating in Grid VOs and
provide services


Expose low level capabilities to mobile application
developers




Linking technology to society


Advanced way for empowering the collaboration of peer
organisations


Defining layered languages and tools


Semantic
-
based data processing and user interaction in
highly competitive businesses environments




Trust, security and dependability


Protection of data and transactions at all levels for a
secure “Future Internet of Services”


Authentication, confidentiality and integrity to enable
accessing to rich multimedia content




Business process collaboration


Innovative and advanced service
-
based solutions for virtualisation of
enterprises


Dynamic set
-
up of new working
environments

Further Information

GREDIA Web Site under the



www.gredia.eu

Questions?