# PVM and MPI

Λογισμικό & κατασκευή λογ/κού

1 Δεκ 2013 (πριν από 4 χρόνια και 6 μήνες)

124 εμφανίσεις

PVM
and

MPI

What is more preferable?

Comparative analysis of PVM and
MPI for the development of physical
applications on parallel clusters

Ekaterina Elts

Assoc. Prof. A.V. Komolkin

Introduction

Computational
Grand
Challenge

problems

Parallel

processing

the

method

of

having

many

small

to

solve

one

large

problem

Two

major

trends

:

MPPs

-

(massively

parallel

processors)

but

cost

\$\$\$

!

distributed

computing

Introduction

The

hottest

trend

today

is

PC

clusters

running

Linux

Many Universities and
companies can afford
16 to 100 nodes.

PVM

and

MPI

are

the

most

used

tools

for

parallel

programming

Contents

Parallel Programming

A Parallel Machine Model

Cluster

A Parallel Programming Model

PVM and MPI

Background

Definition

A Comparison of Features

Conclusion

A Sequential Machine Model

The

von Neumann
computer

A

central

processing

unit

(CPU)

executes

a

program

that

performs

a

sequence

of

and

write

operations

on

an

attached

memory

SISD

S
ingle

I
nstruction

Stream

S
ingle
D
ata

Stream

A Parallel Machine Model

Interconnect

The cluster

A

node

can

communicate

with

other

nodes

by

sending

and

receiving

messages

over

an

interconnection

network

The

von Neumann
computer

MIMD

M
ultiple

I
nstruction

Stream

M
ultiple
D
ata

Stream

A Parallel Programming
Model

input

output

input

output

Parallel algorithm

Sequential (serial) algorithm

Example: scalar product of
vectors

input

output

input

output

Parallel algorithm

Sequential (serial) algorithm

b
a

,
do

i=1,N

S=s+a
i
b
i

enddo

print

S

b
a

,
do

i=1,N/2

s
1
=s
1
+a
i
b
i

enddo

do

i=N/2+1,N

s
2
=s
2
+a
i
b
i

enddo

print

S

S=s
1
+s
2

A Parallel Programming
Model

Message Passing

1

2

3

4

5

0

detailed picture of

Many small tasks solve one large problem

Instantaneous state of computation

Each processor in a message passing
program runs a separate process (sub
-

written in a conventional sequential language

all variables are private

communicate via special subroutine calls

Messages

Messages are packets of data moving
between processes

The message passing system has to be
told the following information
:

Sending process

Source location

Data type

Data length

Receiving process(es)

Destination location

Destination size

Message Passing

SPMD

Single Program Multiple Data

Same program runs
everywhere

Each process only
knows and operates
on a small part of
data

MPMD

Multiple Program Multiple Data

Each process
perform a different
function (input,
problem setup,
solution, output,
display)

What is Master/Slave principle?

The master has the control over the
running application, it controls all data and
it calls the slaves to do there work

PROGRAM

IF (process =
master
) THEN

master
-
code

ELSE

slave
-
code

ENDIF

END

Simple Example
SPMD&Master/Slave

b
a

,

2
2
i
i
b
a
s

1
1
i
i
b
a
s
master

slave

S=
s
1
+s
2

slave

3
3
i
i
b
a
s
For

i

from

rank

step

size

to

N

do

s=s+a
i
b
i

enddo

a
1
b
1
+a
1+size
b
1+size
+a
1+2*size
b
1+2*size
+…

PVM

and

MPI

Background

PVM

The

development

of

PVM

started

in

summer

1989

at

Oak

Ridge

National

Laboratory

(
ORNL
)
.

PVM

was

effort

of

a

single

research

group,

allowing

it

great

flexibility

in

design

of

this

system

MPI

The

development

of

MPI

started

in

April

1992
.

MPI

was

designed

by

the

MPI

Forum

(a

diverse

collection

of

implementors,

library

writers,

and

end

users)

quite

independently

of

any

specific

implementation

1989

94

90

96

97

99

2000

PVM
-
1

MPI
-
1

MPI
-
2

PVM
-
2

PVM
-
3

PVM
-
3.4

PVM

and

MPI

Goals

PVM

A distributed
operating system

Portability

Heterogeneity

Handling

communication

failures

MPI

A

library

for

writing

application

program,

not

a

distributed

operating

system

portability

High

Performance

Heterogeneity

Well
-
defined

behavior

Note:

implementation

specification!

MPI implementations: LAM, MPICH,…

What is MPI ?

MPI
-

Message Passing Interface

A

fixed

set

of

processes

is

created

at

program

initialization,

one

process

is

created

per

processor

mpirun

np

5

program

Each

process

knows

its

personal

number

(rank)

Each

process

knows

number

of

all

processes

Each

process

can

communicate

with

other

processes

Process

can’t

create

new

processes

(in

MPI
-
1
)

What is PVM ?

PVM
-

Parallel Virtual Machine

Is

a

software

package

that

allows

a

heterogeneous

collection

of

workstations

(host

pool)

to

function

as

a

single

high

performance

parallel

machine

(virtual)

PVM,

through

its

virtual

machine

provides

a

simple

yet

useful

distributed

operating

system

It

has

daemon

running

on

all

computers

making

up

the

virtual

machine

PVM Daemon (pvmd)

UNIX process which oversees the operation of
user processes within a PVM application and
coordinates inter
-
machine PVM communications

The pvmd serves as a message router and
controller

One

pvmd

runs

on

each

host

of

a

virtual

machine

The

first

pvmd

(started

by

hand)

is

designated

the

master
,

while

the

others

(started

by

the

master)

are

called

slaves

Only

the

master

can

start

new

slaves

and

them

to

configuration

or

delete

slave

hosts

from

the

machine

master

Executing user

computation

Executing

PVM system

routines

What is Not Different?

Portability

source code written for one architecture
can be copied to a second architecture, compiled and
executed without modification (to some extent)

Support
MPMD

programs as well as
SPMD

Interoperability

the ability of different

implementations
of the same specification to exchange messages

Heterogeneity (to some extent)

PVM & MPI are systems designed to provide users with
libraries for writing portable, heterogeneous, MPMD
programs

Heterogeneity

Architecture

Data format

Computational speed

dynamic

static

Heterogeneity
:
MPI

Different datatypes can be encapsulated in
a single derived type, thereby allowing
communication of heterogeneous
messages. In addition, data can be sent
from one architecture to another
with

data
conversion in heterogeneous networks

(big
-
endian, little
-
endian)
.

Heterogeneity
:
PVM

The PVM system supports heterogeneity
in terms of machines, networks, and
applications. With regard to message
passing, PVM permits messages
containing more than one datatype to be
exchanged between machines having
different data representations.

Process control

-
Ability to start and stop tasks, to find out which
tasks are running, and possibly where they are
running.

PVM

contains all of these capabilities

MPI
-
1

has no defined method to start new task.
MPI
-
2

contain functions to start a group of tasks
and to send a kill signal to a group of tasks

Resource Control

PVM

is inherently
dynamic

in nature, and it
has a rich set of resource control
functions. Hosts can be added or deleted

fault tolerance

efficiency

MPI

is specifically designed to be
static

in
nature to improve performance

Virtual topology

-

only for

MPI

Convenient process naming

Naming scheme to fit the communication pattern

Simplifies writing of code

Can allow MPI to optimize communications

Virtual topology example

A

virtual

topology

of

twelve

processes

-

grid

with

a

cyclic

boundary

condition

in

one

direction

e
.
g
.

processes

0

and

9

are

``connected''
.

The

numbers

represent

the

rank

and

the

conceptual

coordinates

mapped

to

the

ranks

Message Passing operations

MPI
: Rich message support

PVM
: Simple message passing

Point
-
to
-
Point communications

A synchronous
communication does not
complete until the message
.

An asynchronous
communication completes as
soon as the message is on
its way

Non
-
blocking operations

Non blocking communication allows useful work to be performed
while waiting for the communication to complete

Collective communications

Barrier

A barrier operation
synchronises a
number of
processors.

message to a number of
recipients

Reduction
operations

Reduction operations
reduce data from a
number of processors
to a single item.

Fault Tolerance:
MPI

MPI
standard is based on a
static model

If a member of a group failed

for some
reason, the specification mandated that
rather than continuing which would lead to
unknown results in a doomed application,
the group is invalidated and
the application
halted

in a clean manner.

In simple if something fails, everything
does.

Fault Tolerance:
MPI

Fault Tolerance:
MPI

Failed Node

There is a failure and…

Fault Tolerance:
MPI

Failed Node

… the application is shut down

Fault Tolerance:
PVM

PVM supports a basic fault notification scheme:
it doesn’t automatically recover an application
after a crash, but it does provide notification
primitives to allow fault
-
tolerant applications to
be built

The
Virtual Machine

is dynamically
reconfigurable

A pvmd can recover from the loss of any foreign
pvmd except the master. The master must
never crash

Fault Tolerance:
PVM

Virtual Machine

Fault Tolerance:
PVM

Virtual Machine

Failed Node

Fault Tolerance:
PVM

Virtual Machine

Fast host delete or recovery from fault

Conclusion

PVM

Virtual machine concept

Simple message passing

Communication topology
unspecified

Interoperate across host
architecture boundaries

Portability over performance

Resource and process
control

Robust fault tolerance

MPI

No such abstraction

Rich message support

Support logical communication
topologies

Some realizations do not
interoperate across architectural
boundaries

Performance over flexibility

Primarily concerned with
messaging

More susceptible to faults

Each API has it’s unique strengths

Conclusion

PVM is better for:

Heterogeneous cluster,
resource and process
control

The size of cluster and
the time of program’s
execution are great

MPI is better for:

Supercomputers (
PVM is
not supported
)

Application for MPP

Max performance

Application needs rich
message support

Each API has it’s unique strengths

Acknowledgments