Recent Advances in Grid Computing and Business Models: A Gridbus Perspective

scarfpocketAI and Robotics

Oct 24, 2013 (3 years and 7 months ago)

54 views

Recent Advances in Grid Computing
and Business Models: A Gridbus
Perspective

Rajkumar Buyya

Gri
d and
D
istributed
S
ystems (GRIDS) Laboratory

Dept. of Computer Science and Software Engineering

The University of Melbourne

Melbourne, Australia

www.gridbus.org

WW Grid

Grid Business Symposium 2005, Seoul, Korea

2

Outline


Introduction


Utility Networks and Grid Computing


Global Grids and Challenges


Grid Initiatives


World
-
wide with Australia and India Perspective


Introduction to Gridbus Project and Grid
Economy


Grid Service Broker


Architecture, Design and Implementation


Performance Evaluation: Experiments in Creation
and Deployment of Applications on Global Grids


A Case Study in High Energy Physics


Economy
-
based Scheduling in Data Grids


Summary

3

4 Essential Utilities and

Delivery Networks

(1) Water

(2) Electricity

(3) Gas

(4) Telephone

4

(5) IT services as the fifth utility

(water, electricity, gas, telephone, IT)

eScience

eBusiness

eGovernment

eHealth

Multilingual

eEducation



5

A Bird Eye View of

World
-
Wide Grid Environment

Grid Resource Broker

Resource Broker

Application

Grid Information Service

Grid Resource Broker

database

R
2

R
3

R
N

R
1

R
4

R
5

R
6

Grid Information Service

7

Grid Challenges

Security

Resource Allocation

& Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Construction

8

Some Grid Initiatives Worldwide


Australia


Nimrod
-
G


Gridbus


DISCWorld


GrangeNet.


APACGrid


ARC eResearch


Brazil


OurGrid, EasyGrid


LNCC
-
Grid + many others


China


ChinaGrid


Education


CNGrid
-

application


Europe


UK eScience


EU Grids..


and many more...


India


I
-
Grid


Japan


NAGERI



Korea...

N*Grid


Singapore

NGP


USA


Globus



NASA IPG



AccessGrid


TeraGrid


Cyberinfrasture


Industry Initiatives


IBM On Demand Computing


HP Adaptive Computing


Sun N1


Microsoft
-

.NET


Oracle 10g


Satyam


Grid Practice


Infosys, Wipro, TCS


StorageTek

Grid..


Public Forums


Global Grid Forum


Australian Grid Forum



Conferences:


CCGrid


Grid


HPDC


E
-
Science

http://www.gridcomputing.com

1.3 billion


3 yrs

1 billion


5 yrs

450million


5 yrs

486million


5 yrs

1.3 billion (Rs)

27 million

2? billion

120million


5 yrs

10

Grid Computing in Australia

(Courtesy: Jihyoun Park, SNU Visitor to Melbourne)

Academia

Government

Collaboration

Indus
try

11

Academic activities

1 University laboratories for Grid computing


-

Uni. of Melbourne(GRIDS lab): Gridbus (GridSim, GMD, GridBank, Alchemi, ..),
Master of Engineering in Distributed Computing




-

Monash Uni.: GriddlsS (Legacy SW to the computational grid), Nimrod
-
G


-

Australian national Uni. (Internet Futures Group)


-

Sydney Uni.(ViSLAB): high performance visualization &computing


-

Uni. of Adelaide (DHPC Group): DISCWorld


-

Queensland Uni. of Technology (PLAS): G2 (.NET based)


2 Grid Infrastructure Projects


APACGrid, National Neurosciece Facility, Australian Virtual Observatory, several
state level facilities (VPAC, TPAC, SAPAC, QPSF, IVEC)


3 Grid Applications


* Asia Pacific Bioinformatics Network/ Virtual Drug Design: Molecular Modeling for
Drug Design on P2P Grid/ HEPGrid: High Energy Physics and the Grid Network/
Access Grid/Australian Computational Earth Systems Simulator/.


* Recently 30 more applications are funded as part of ARC e
-
Research


* Govt. has formed “National e
-
Research Coordination Committee”.


12

Grid Computing in India

Academia

Governm
ent

Collabor
ation

Industry

(majority focus on

Grid integration)


13

Grid Computing in India: Academic and
Industrial Activities


Academic and Government Initiatives:


TIFR, IITM, Anna University, IITD, UoH, etc.


C
-
DAC

s Garuda


Ministry of IT


Software Companies in India:


Top 4 Indian IT Companies: Satyam, Infosys, TCS (Tata
Consultancy Service), and Wipro.


Oracle 10g, IBM, HP, Sun ertc. have a large Grid
development centers in Bangalore, India.


Satyam is leading the pack in Grid Business push:


Grid Practice Centre with top management support.


Singned MoU with Melbourne University and extensively
using Gridbus in powering applications.


Also contributing the development of Gridbus technologies
(e.g., Alchemi)


SEI CMM Level 5 principles.


Application Verticals: Manufacturing, Security, Life
Sciences, Finance


14

15

Australian and Indian Grid Efforts
Compared

Australia

India

Korea: Is it like Australia or India?

16

The Gridbus Project @ Melbourne:

Enable Leasing of ICT Services on Demand

WWG

World Wide Grid!


On Demand Utility Computing

Gridbus

Distributed Data

17

The Gridbus Project:
http://www.gridbus.org


A multi
-
institutional

Open Source


R&D Project with focus on:


Architecture, Specification, and Open Source Reference Implementation.


Service
-
Oriented Grid, Utility Computing & Distributed Data and Computation Economy


Scaling from Desktops, Clusters, Cluster Federation, Enterprise Grids to Global Grids.


Alchemi: Harnessing .NET/Windows
-
based Resources


Grid Market Directory and Web Services


Grid Bank: Accounting and Transaction Management


Visual Tools for Creation of Distributed Applications


Workflow Composition and Deployment Services


Data Grid Brokering and Grid Economy Services


Data Replication Strategies


GridSim Toolkit: Enhanced to support Data Grid, Reservation, etc.


Libra: SLA
-
based Allocation of Cluster Resources


Coupling of Clusters and Computational Economy


WWG: Global Data Intensive Grid Testbed


Application Enabler Projects:


High
-
Energy Physics , Astronomy, Brain Activity Analysis


Osaka U., Natural Language
Processing, Portfolio Analysis


Spain, BioGrid
-

WEHI (via APACGrid), SensorGrid
(NICTA), Medical Imaging (HFI)


Supported by:

18

Grid Economy: Methodology for Sustained
Resourced Sharing and Managing

Supply
-
and
-
Demand for Resources

20

Grid Entities and Architecture

GSP site
scheduler

accounting

Grid consumer

Market

Maker

GSP
global
scheduler

broker

Resource
owners

End users

Private enterprises

National providers

GSP site
scheduler

Resource
owners

21

Grid Node N

A Reference Service
-
Oriented

A
rchitecture for Utility Grids

Grid Consumer

Programming

Environments

Grid Resource Broker

Grid Service Providers

Grid Explorer

Schedule Advisor

Trade Manager

Job
Control

Agent

Deployment Agent

Trade Server

Resource Allocation

Resource

Reservation

R
1

Misc. services

Information
Service

R
2

R
m



Pricing
Algorithms

Accounting

Grid Node1

Grid Middleware

Services



Health

Monitor

Grid Market
Services

JobExec

Info ?

Secure

Trading

QoS

Storage

Sign
-
on

Grid Bank

Applications

Data Catalogue

22

Gridbus and Complementary
Technologies


realizing Utility Grid

AIX

Solaris

Windows

Linux

.NET

Grid

Fabric

Software

Grid

Applications

Core Grid

Middleware

User
-
Level

Middleware

(Grid Tools)

Grid

Bank

Grid

Exchange &
Federation

JVM

Grid Brokers:

X
-
Parameter Sweep Lang.

Gridbus Data Broker

MPI

Condor

SGE

Tomcat

PBS

Alchemi

Workflow

IRIX

OSF1

Mac

Libra

Globus

Unicore



Grid
Market

Directory

PDB

CDB

Worldwide Grid

Grid

Fabric

Hardware



Portals

Science

Commerce

Engineering



Collaboratories



Workflow Engine

Grid
Storage
Economy

Grid Economy

NorduGrid

XGrid

ExcellGrid

Nimrod
-
G


G
R
I
D
S
I
M


Gridscape

23

Alchemi: .NET
-
based Enterprise Grid
Platform & Web Services


Internet


Internet

Alchemi Worker Agents

Alchemi Manager

Alchemi Users


Web

Services


SETI@Home like Model


General Purpose


Dedicated/Non
-
dedicate
workers


Role
-
based Security


.NET and Web Services


C# Implementation


GridThread and Job Model
Programming


Easy to setup and use



Widely in use!

24

Some Users of Alchemi

Tier Technologies
, USA

Large scale document processing using Alchemi framework

CSIRO
, Australia

Natural Resource Modeling

The Friedrich Miescher Institute (FMI) for Biomedical Research
, Switzerland

Patterns of transcription factors in mammalian genes

Satyam Computers Applied Research Laboratory
, India

Micro
-
array data processing using Alchemi framework

The University of Sao Paulo
, Brazil

The Alchemi Executor as a Windows Service

stochastix GmbH
, Germany

Asynchronous Excel Tasks using ManagedXLL and Alchemi .Net Grid
Computing framework.

Many users in Universities: See next for an example.

25

On Demand Assembly of Services: Putting
Them All Together

Data Source

(Instruments/dis
tributed sources)

Data Replicator

(GDMP)

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP

(Accounting Service)

Gridbus

GridBank

Data

GSP

(e.g., UofM)

PE

GSP

(e.g., VPAC)

PE

GSP

(e.g., IBM)

CPU

or

PE

Grid Service (GS)

(Globus)

Alchemi

GS

GTS

Cluster Scheduler

Grid Service Provider (GSP)


(e.g., CERN)

PE

Cluster Scheduler

8

Grid

Resource Broker

2

Visual Application Composer

Application Code

Explore

data

1

3

6

4

5

9

7

10

11

Bill

12

Data Catalogue

The Gridbus Grid Service
Broker for Data Grid
Applications

Builds on the Nimrod
-
G Computational
Grid Broker and Computational Economy
[Buyya, Abramson, Giddy, Monash
University, 1999
-
2001]

And

Extends its notion for Data and Service
Grids

27

Gridbus Broker Architecture

Grid Middleware

Gridbus Client

Gridbus Client

Gribus Client

Grid Info Server

Schedule Advisor

Trading Manager

Gridbus Farming Engine

Record

Keeper

Grid Explorer

GE GIS, NWS

TM TS

RM & TS

Grid Dispatcher

RM: Local Resource Manager, TS: Trade Server

G

G

C

U

Globus enabled node.

A

L

Alchemi enabled node.

(Data Grid Scheduler)

Data

Catalog

Data

Node

Unicore enabled node.

$

$

$

App, T, $, Opt

(Bag of Tasks Applications)

29

Gridbus Services for eScience
applications


Application Development Environment:


XML
-
based language for composition of task farming
(legacy) applications as parameter sweep applications.


Task Farming APIs for new applications.


Web APIs (e.g., Portlets) for Grid portal development.


Threads
-
based Programming Interface


Workflow interface and Gridbus
-
enabled workflow
engine.


Resource Allocation and Scheduling


Dynamic discovery of optional computational and data
nodes that meet user QoS requirements.


Hide L
ow
-
Level Grid Middleware interfaces


Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.

30

Figure 3 : Logging into the portal.

Drug Design

Made Easy!

Click Here for Demo

Economy
-
based Data Grid
Scheduling

High Energy Physics as eScience
Application Case Study

CLICK HERE TO SKIP IF
RUNNING OUT of TIME

32

Australian Belle Data Grid Testbed

33

Case Study: Event Simulation and
Analysis

B0
-
>D*+D*
-
Ks




Simulation and Analysis Package
-

Belle Analysis Software Framework (BASF)



Experiment in 2 parts


Generation of Simulated Data and Analysis of the
distributed data




Analyzed 100 data files (30MB each) were distributed among the five nodes

34

Resources Used and their Service Price

Organization



Node details

Role

Cost (in G$/CPU
-
sec)

CS,UniMelb

belle.cs.mu.oz.au

4 CPU, 2GB RAM, 40 GB HD,
Linux

Broker host, Data
host, NWS server

N.A. (Not used as a
compute resource)

Physics, UniMelb

fleagle.ph.unimelb.edu.au

1 CPU, 512 MB RAM, 40 GB HD,
Linux

Replica Catalog
host, Data host,
Compute resource,
NWS sensor

2

CS, University of
Adelaide

belle.cs.adelaide.edu.au

4 CPU (only 1 available) , 2GB
RAM, 40 GB HD, Linux

Data host, NWS
sensor

N.A. (Not used as a
compute resource)

ANU, Canberra

belle.anu.edu.au

4 CPU, 2GB RAM, 40 GB HD,
Linux

Data host, Compute
resource, NWS
sensor

4

Dept of Physics,
USyd

belle.physics.usyd.edu.au

4 CPU (only 1 available), 2GB
RAM, 40 GB HD, Linux

Data host, Compute
resource, NWS
sensor

4

VPAC,
Melbourne

brecca
-
2.vpac.org

180 node cluster (only head node
used), Linux

Compute resource,

NWS sensor

6

35

Network Cost (in Grid $/Currency!)

36

Deploying Application Scenario


A data grid scenario with 100 jobs and
each accessing remote data of ~30MB


Deadline: 3hrs.


Budget: G$ 60K


Scheduling Optimisation Scenario:


Minimise Time


Minimise Cost


Results:


37

Grid and Gridbus Technologies for
Various Grid (Market) Types

commercial

scientific

free trading

regulation

Public

computin
g

(Alchemi)

National provider

(Globus, Gridbus,..)

Private
enterprises

(Libra, Gridbus,
Globus)

Application Category

Sharing

Model

39

Summary and Conclusion


Grids exploit synergies that result from
cooperation of autonomous entities:


Resource sharing, dynamic provisioning, and
aggregation at global level.


Grid Economy provides incentive needed for
sustained cooperation.


Grid Network has potential to serve as
Cyberinfrastructure for Utility Computing


Grids offer enormous opportunities for realizing
eScience and eBusiness at global level.

40

Any Questions ?

Gridbus Project
-

http://www.gridbus.org

41

Thanks for your attention!

The Gridbus Cooperation!

http://www.gridbus.com


Backup Slides

45

What do Grids aim for and how to
support them.


Grids aim at exploiting synergies that result
from cooperation of autonomous distributed
entities. Synergies include:


Resource sharing



On
-
demand


Virtual Enterprises creation


Aggregation of resources on demand.


For this cooperation to be sustainable,
participants needs to have (economic) incentive.


Therefore,

incentive


mechanisms should be
considered as one of key design parameters of
Grid computing.

46

Grid Market (Participant) Types and
Application Category

commercial

scientific

free trading

regulation

Public

computin
g

National provider

Private
enterprises

Application Category

Sharing

Model

47

Appropriate Market Model for
different market types

strong

weak

high

low

Variable
price
auction

Posted
price
oligopoly

Commodity
market

Demand elasticity

Willingn
ess to
Pay

49

Deadline (D) and Budget (B) Constrained
Scheduling Algorithms

Algorithm

Execution
Time (D)

Execution
Cost (B)

Compute
Grid

Data Grid

Cost Opt

Limited by D

Minimize

Yes

Yes

Cost
-
Time
Opt

Minimize if
possible

Minimize

Yes

Time Opt

Minimize

Limited by B

Yes

Yes

Conservative
-
Time

Opt

Minimize

Limited by B,
jobs have
guaranteed
minimum
budget

Yes