Cluster Programming Technology and its Application in Meteorology

coleslawokraΛογισμικό & κατασκευή λογ/κού

1 Δεκ 2013 (πριν από 3 χρόνια και 8 μήνες)

147 εμφανίσεις

Cluster Programming Technology
and its Application in Meteorology

Computer and Automation

Research Institute

Hungarian Academy of Sciences

Hungarian

Meteorological

Service

Silicon
Computers Ltd.

Background


Hungarian Meteorological Service

developed


MEANDER

(MEsoscale Analysis Nowcasting and DEcision Routines)


crucial task in the protection of life and property (storm warning
at Lake Balaton, weather warnings for aviation...)


based on incoming meteorology information and
computational
intensive methods



MTA SZTAKI

developed



P
-
GRADE

parallel programming environment


efficient, graphical

support for the entire life cycle of parallel
program development



Cluster programming technology and its applications in meteorology


joint

project

of

MTA

SZTAKI
,

Hungarian

Meteorological

Service

and

Silicon

Computers

Ltd
.


supported

by

Research

&

Development

Division
,

Ministry

of

Education





Goal
:

Analysis

of

all

the

available

meteorology

information



producing

parameters

on

a

high

resolution

regular

mesh



(
10
km
--
>

1
km)



ultra
-
short

range

forecast

(up

to

6

hours)



Application
: Forecasting dangerous weather situations (storms,
fog, etc.)



Meteorology

information
:

surface

level

measurements,

high
-
altitude

measurements,

radar,

satellite,

lightning,

results

of

previous

computed

models,

etc
.




Basic

parameters
:

pressure,

temperature,

humidity,

wind,





Derived

parameters
:

type

of

clouds,

visibility,




MEANDER Program Package

Structure of MEANDER

First guess data

ALADDIN

SYNOP
data

Satellite

data

Radar

data


CANARI

DELTA
analysis

Basic fields:
pressure,
temperature,
humidity, wind

Derived fields:

Type of clouds,
visibility, etc.


BASIC GRID

Radar
to
grid

Satellite
to grid


“Present

weather

Type of
clouds


Overcas
t


Visibilit
y

Rainfall
phase

Visualization

For
meteorologist
:HAWK

For users:
GIF

Lightning

decode

36000 km

Satellite raw image

Receiving

image


Transformation &

Interpolation &


Processing

Processing:

altitude of clouds

Transformation

Interpolation to basic GRID

Processing of satellite images

THE PROBLEM


MEANDER

Sequential

code


C, C++


Fortran


MEANDER

Parallel

version


PC claster

SGI Origin 2
000

SUN E10000


Parallelisation
?

Debugging
?

Performance
?

GRAPNEL

g
raphical

l
anguage

/
GRP2C

pre
-






compiler

Unix
-
like o
perating
s
ystem
s
&


C, C++, Fortran p
rogramming
l
ibraries

PVM
or MPI m
essage
p
assing
l
ibrary

DIWIDE

distributed


debugger

GRED

g
raphical

editor

GRM

monitor


d
istributed


c
heckpointing

tool



GRP_CHKPT

GRP_MM



m
igration module

GRP_LB

l
oad
b
alancer

PROVE

p
erformance

visualization

tool

TLC


model

c
hecker

Macrostep

debugger

The solution:

P
-
GRADE
development environment

Results:
temperature and
wind at 850 hPa
level

MEANDER: 3D FIELDS

2D
analysis

Radar

data

Satellite

data

Delta 3D

analy
s
is


...computes the basic
meteorological fields:



pressure, temperature,
humidity, wind velocity
and direction



for a high resolution 3D
mesh (
10km
-
1km)

Implementation
of

DELTA analysis in

P
-
GRADE

Fortran

seq. code

P
-
GRADE version of MEANDER

for clusters & supercomputers

25 x

4

x

10

x

25 x

25 x

2
0

x

Live demo

(5
th

DataGRID conference)

34
Mbit

Shared

PERL
-
GRID

CONDOR
-
PVM

job







11
/5
Mbit

Dedi
cated

P
-
GRADE

PERL
-
GRID

job

ftp.met.hu




netCDF


512 kbit

Shared

netCDF

input

Parallel execution and GRM

GRM

TRACE &

Results

GRM

TRACE &

Results

PROVE performance visualization

Edit,
debugging

Performance
-
analysis

Testing,

Execution

Resource requirements of P
-
GRADE

Advantages of P
-
GRADE environment



Efficient support

for each stage of parallel
program development



Fast parallelisation

of existing algorithms



Reusability

of sequential code



Hiding

of low level communication functions



Unified and integrated

graphical concept



Predefined communication
templates



Support for
hierarchical design



Even
non
-
professional programmers can use it


(steep learning curve)



Portability

from supercomputers to
PC clusters


Support for cluster computing

More information
:
www.lpds.sztaki.hu

New facilities in P
-
GRADE

for long
-
running parallel

applications:




Distributed
checkpointing



Process
migration



Dynamic
load balancing



Fault
-
tolerance

execution

I. Execution without

load balancing

III. Load balancing

& migration

IV. Result:

Faster

execution

II. Checkpointing