Performance guided scheduling in GENIE through ICENI

nutmegactΛογισμικό & κατασκευή λογ/κού

10 Νοε 2012 (πριν από 8 χρόνια και 8 μήνες)

334 εμφανίσεις

Performance guided scheduling
in GENIE through ICENI

http://www.genie.ac.uk/


2

Contents

1.
What is GENIE?

2.
Previous work


Grid infrastructure

3.
Limitations of present infrastructure

4.
Introduction to ICENI

5.
Performance experiments

6.
Summary and conclusions

7.
Future work

3

What is GENIE?



G
rid
EN
abled
I
ntegrated
E
arth system model.


Investigate long term changes to the Earth’s climate (i.e.
global warming) by integrating numerical models of the
Earth system.


e
-
Science aims:


Flexibly couple together state
-
of
-
the
-
art components
to form unified Earth System Model (ESM).


Execute resultant ESM on a Grid infrastructure.


Share resultant data produced by simulation runs.


Provide high
-
level open access to the system,
creating and supporting virtual organisation of Earth
System modellers.

4

GENIE model components

GENIE
www.genie.ac.uk
Grid ENabled Integrated
Earth system model
open ocean
marine biota
sea ice
ice
sheets
land surface and hydrology
deep
ocean
terrestrial
biota
atmosphere
soils
coastal seas
5

Previously in GENIE…

Investigating the thermohaline circulation


Investigated influence of
freshwater transport upon
global ocean circulation


Performed several parameter
sweep experiments, each
consisting of ~1000
simulations of a GENIE
prototype.


Used a Grid infrastructure:

i.
Portal


create, submit and manage experiments.

ii.
Condor pool


execute simulations in parallel.

iii.
Database management system


archive and
process resultant data.

6

Previously in GENIE…

Scientific achievements

Intensity of the thermohaline circulation as a
function of freshwater flux between Atlantic
and Pacific oceans and mid
-
Atlantic and
North Atlantic.

Surface air temperature difference between
extreme states (off
-

on) of the thermohaline
circulation.

North Atlantic 2

C colder when the
circulation is off.

New scientific findings


papers published!

7

Limitations of Grid infrastructure


GENIE model hard
-
coded


cannot use
alternative models without recoding.


Format of input/output data not flexible.


Parameter space being investigated is fixed.


True resource brokering not taking place.

Solution:

8

Advantages of ICENI


ICENI Netbeans client allows experiment to
be built in a systematic and repeatable way.


Component based programming model
provides flexibility and extensibility.


Service oriented architecture allows for true
resource brokering and Grid enablement of
application.

9

ICENI Binary Component


Allows you to wrap a binary executable as an
ICENI component.

binary
executable

JDML

stdin

files

stdout

stderr

10

A GENIE experiment as an ICENI
application


setup
component

GENIE

binary
component

archive
component

GENIE

binary
component

GENIE

binary
component

splitter
component

collator
component

11

A GENIE experiment as an ICENI
application


Introduce high
-
throughput resource
launcher…

setup
component

archive
component

GENIE

binary
component

resource
launcher

Sun Grid
Engine

Condor
pool

Bash

LAUNCHES

TO

12

Performance experiments


Ran 8 different types of experiments to evaluate
performance of ICENI:









Each experiment run several times in order to obtain an
average sojourn time.

#

Resource

Job Type

Middleware

1

Solaris

Bash

Non
-
ICENI

2

ICENI

3

Condor

Non
-
ICENI

4

ICENI

5

Linux

Bash

Non
-
ICENI

6

ICENI

7

Condor

Non
-
ICENI

8

ICENI

Solaris



Shared memory server



8


900MHz UltraSparc II CPUs



16Gb memory


Linux:



Beowulf cluster



16


2GHz Intel Dual Xeon CPUs



2Gb memory

13

Performance results


2.33
2.5
2.44
11.46
2.51
2.52
2.49
11.65
0
2
4
6
8
10
12
Bash
Condor
Bash
Condor
Solaris
Linux
Resource
Time (hours)
Non-ICENI
ICENI
14

Performance results


2.33
2.5
2.44
11.46
2.51
2.52
2.49
11.65
0
2
4
6
8
10
12
Bash
Condor
Bash
Condor
Solaris
Linux
Resource
Time (hours)
Non-ICENI
ICENI
Under Solaris
Condor is ~5 times
faster than Bash



sequential vs
parallel execution.

15

Performance results


2.33
2.5
2.44
11.46
2.51
2.52
2.49
11.65
0
2
4
6
8
10
12
Bash
Condor
Bash
Condor
Solaris
Linux
Resource
Time (hours)
Non-ICENI
ICENI
Under Linux
Condor and Bash
perform similarly


only 2 nodes in
Linux Condor pool.

16

Performance results


2.33
2.5
2.44
11.46
2.51
2.52
2.49
11.65
0
2
4
6
8
10
12
Bash
Condor
Bash
Condor
Solaris
Linux
Resource
Time (hours)
Non-ICENI
ICENI
ICENI introduces
scheduling and
deployment
overhead to the
sojourn time.

17

Summary


Shown how ICENI middleware can be used to
launch GENIE experiments.


Performance overhead is insignificant when
compared to advantages of using ICENI to deploy
jobs.


Can create and schedule ensemble experiments
across multiple computational resources using
true resource brokering.

18

Future work


Need to repeat experiments with Sun Grid
Engine launcher.


Need to incorporate component based
GENIE model in experiments.


ICENI is evolving…


Adopt new web services.


Decouple into separate functionalities.


Use GridSAM to launch experiments.


(please visit the LeSC booth for a demo)

19

Acknowledgments



GENIE investigators:


Prof. Paul Valdes (Bristol), Prof. John Shepherd (SOC,
Southampton), Prof. Andrew Watson (UEA), Prof.
Melvyn Cannell (CEH Edinburgh), Dr. Anthony Payne
(Bristol), Prof. Richard Harding (CEH Wallingford), Prof.
Simon Cox (SReSC), Dr. Steven Newhouse (OMII) and
Prof. John Darlington (LeSC).


Recognised researchers:


Dr. Stephen McGough (LeSC), Andrew Yool (SOC), Dr.
Robert Marsh (SOC), Dr. Timothy Lenton (UEA) and Dr.
Neil Edwards (Bern).