PVM and MPI

footballsyrupSoftware and s/w Development

Dec 1, 2013 (3 years and 8 months ago)

77 views

PVM
and

MPI

What is more preferable?


Comparative analysis of PVM and
MPI for the development of physical
applications on parallel clusters

Ekaterina Elts

Scientific adviser:
Assoc. Prof. A.V. Komolkin

Introduction


Computational
Grand
Challenge

problems


Parallel

processing



the

method

of

having

many

small

tasks

to

solve

one

large

problem



Two

major

trends

:


MPPs

-

(massively

parallel

processors)



but

cost

$$$

!


distributed

computing

Introduction


The

hottest

trend

today

is

PC

clusters

running

Linux



Many Universities and
companies can afford
16 to 100 nodes.


PVM

and

MPI

are

the

most

used

tools

for

parallel

programming


Contents


Parallel Programming



A Parallel Machine Model


Cluster


A Parallel Programming Model


Message Passing Programming Paradigm


PVM and MPI





Background


Definition



A Comparison of Features


Conclusion







A Sequential Machine Model

The

von Neumann
computer


A

central

processing

unit

(CPU)

executes

a

program

that

performs

a

sequence

of

read

and

write

operations

on

an

attached

memory

SISD


S
ingle

I
nstruction

Stream



S
ingle
D
ata

Stream

A Parallel Machine Model

Interconnect



The cluster

A

node

can

communicate

with

other

nodes

by

sending

and

receiving

messages

over

an

interconnection

network

The

von Neumann
computer

MIMD


M
ultiple

I
nstruction

Stream



M
ultiple
D
ata

Stream

A Parallel Programming
Model



input

output

input

output

Parallel algorithm

Sequential (serial) algorithm


Example: scalar product of
vectors



input

output

input

output

Parallel algorithm

Sequential (serial) algorithm

b
a


,
do

i=1,N


S=s+a
i
b
i

enddo

print

S

b
a


,
do

i=1,N/2


s
1
=s
1
+a
i
b
i

enddo

do

i=N/2+1,N


s
2
=s
2
+a
i
b
i

enddo

print

S

S=s
1
+s
2

A Parallel Programming
Model



Message Passing

1

2

3

4

5

0

detailed picture of

a single task

Many small tasks solve one large problem

Instantaneous state of computation

Message Passing Paradigm


Each processor in a message passing
program runs a separate process (sub
-
program, task)



written in a conventional sequential language


all variables are private


communicate via special subroutine calls

Messages


Messages are packets of data moving
between processes


The message passing system has to be
told the following information
:


Sending process


Source location


Data type


Data length


Receiving process(es)


Destination location


Destination size

Message Passing

SPMD

Single Program Multiple Data



Same program runs
everywhere


Each process only
knows and operates
on a small part of
data

MPMD

Multiple Program Multiple Data



Each process
perform a different
function (input,
problem setup,
solution, output,
display)

What is Master/Slave principle?


The master has the control over the
running application, it controls all data and
it calls the slaves to do there work


PROGRAM


IF (process =
master
) THEN




master
-
code


ELSE




slave
-
code


ENDIF


END

Simple Example
SPMD&Master/Slave

b
a


,


2
2
i
i
b
a
s


1
1
i
i
b
a
s
master

slave

S=
s
1
+s
2

slave



3
3
i
i
b
a
s
For

i

from

rank

step

size

to

N

do


s=s+a
i
b
i

enddo

a
1
b
1
+a
1+size
b
1+size
+a
1+2*size
b
1+2*size
+…

PVM

and

MPI

Background

PVM


The

development

of

PVM

started

in

summer

1989

at

Oak

Ridge

National

Laboratory

(
ORNL
)
.


PVM

was

effort

of

a

single

research

group,

allowing

it

great

flexibility

in

design

of

this

system

MPI


The

development

of

MPI

started

in

April

1992
.


MPI

was

designed

by

the

MPI

Forum

(a

diverse

collection

of

implementors,

library

writers,

and

end

users)

quite

independently

of

any

specific

implementation


1989

94

90

96

97

99

2000

PVM
-
1

MPI
-
1

MPI
-
2

PVM
-
2

PVM
-
3

PVM
-
3.4

PVM

and

MPI

Goals

PVM


A distributed
operating system


Portability


Heterogeneity


Handling

communication

failures


MPI


A

library

for

writing

application

program,

not

a

distributed

operating

system


portability


High

Performance


Heterogeneity


Well
-
defined

behavior

Note:

implementation

specification!

MPI implementations: LAM, MPICH,…

What is MPI ?

MPI
-

Message Passing Interface



A

fixed

set

of

processes

is

created

at

program

initialization,

one

process

is

created

per

processor





mpirun


np

5

program


Each

process

knows

its

personal

number

(rank)


Each

process

knows

number

of

all

processes


Each

process

can

communicate

with

other

processes


Process

can’t

create

new

processes

(in

MPI
-
1
)

What is PVM ?

PVM
-

Parallel Virtual Machine


Is

a

software

package

that

allows

a

heterogeneous

collection

of

workstations

(host

pool)

to

function

as

a

single

high

performance

parallel

machine

(virtual)


PVM,

through

its

virtual

machine

provides

a

simple

yet

useful

distributed

operating

system


It

has

daemon

running

on

all

computers

making

up

the

virtual

machine

PVM Daemon (pvmd)


UNIX process which oversees the operation of
user processes within a PVM application and
coordinates inter
-
machine PVM communications


The pvmd serves as a message router and
controller


One

pvmd

runs

on

each

host

of

a

virtual

machine


The

first

pvmd

(started

by

hand)

is

designated

the

master
,

while

the

others

(started

by

the

master)

are

called

slaves


Only

the

master

can

start

new

slaves

and

add

them

to

configuration

or

delete

slave

hosts

from

the

machine

master

Executing user

computation

Executing

PVM system

routines

What is Not Different?



Portability



source code written for one architecture
can be copied to a second architecture, compiled and
executed without modification (to some extent)


Support
MPMD

programs as well as
SPMD


Interoperability


the ability of different

implementations
of the same specification to exchange messages


Heterogeneity (to some extent)


PVM & MPI are systems designed to provide users with
libraries for writing portable, heterogeneous, MPMD
programs

Heterogeneity


Architecture


Data format


Computational speed


Machine load


Network load


dynamic

static

Heterogeneity
:
MPI


Different datatypes can be encapsulated in
a single derived type, thereby allowing
communication of heterogeneous
messages. In addition, data can be sent
from one architecture to another
with

data
conversion in heterogeneous networks

(big
-
endian, little
-
endian)
.

Heterogeneity
:
PVM


The PVM system supports heterogeneity
in terms of machines, networks, and
applications. With regard to message
passing, PVM permits messages
containing more than one datatype to be
exchanged between machines having
different data representations.

Process control

-
Ability to start and stop tasks, to find out which
tasks are running, and possibly where they are
running.



PVM

contains all of these capabilities




it can spawn/kill tasks dynamically


MPI
-
1

has no defined method to start new task.
MPI
-
2

contain functions to start a group of tasks
and to send a kill signal to a group of tasks

Resource Control


PVM

is inherently
dynamic

in nature, and it
has a rich set of resource control
functions. Hosts can be added or deleted


load balancing


task migration


fault tolerance


efficiency



MPI

is specifically designed to be
static

in
nature to improve performance

Virtual topology


-

only for

MPI


Convenient process naming


Naming scheme to fit the communication pattern


Simplifies writing of code


Can allow MPI to optimize communications

Virtual topology example



A

virtual

topology

of

twelve

processes

-

grid

with

a

cyclic

boundary

condition

in

one

direction

e
.
g
.

processes

0

and

9

are

``connected''
.


The

numbers

represent

the

rank

and

the

conceptual

coordinates

mapped

to

the

ranks

Message Passing operations


MPI
: Rich message support


PVM
: Simple message passing

Point
-
to
-
Point communications

A synchronous
communication does not
complete until the message
has been received
.

An asynchronous
communication completes as
soon as the message is on
its way

Non
-
blocking operations


Non blocking communication allows useful work to be performed
while waiting for the communication to complete


Collective communications

Barrier

A barrier operation
synchronises a
number of
processors.


Broadcast

A broadcast sends a
message to a number of
recipients

Reduction
operations

Reduction operations
reduce data from a
number of processors
to a single item.

Fault Tolerance:
MPI


MPI
standard is based on a
static model



If a member of a group failed

for some
reason, the specification mandated that
rather than continuing which would lead to
unknown results in a doomed application,
the group is invalidated and
the application
halted

in a clean manner.


In simple if something fails, everything
does.


Fault Tolerance:
MPI

Fault Tolerance:
MPI

Failed Node

There is a failure and…

Fault Tolerance:
MPI

Failed Node

… the application is shut down

Fault Tolerance:
PVM


PVM supports a basic fault notification scheme:
it doesn’t automatically recover an application
after a crash, but it does provide notification
primitives to allow fault
-
tolerant applications to
be built


The
Virtual Machine

is dynamically
reconfigurable


A pvmd can recover from the loss of any foreign
pvmd except the master. The master must
never crash


Fault Tolerance:
PVM

Virtual Machine

Fault Tolerance:
PVM

Virtual Machine

Failed Node

Fault Tolerance:
PVM

Virtual Machine

Fast host delete or recovery from fault

Conclusion

PVM


Virtual machine concept


Simple message passing


Communication topology
unspecified


Interoperate across host
architecture boundaries



Portability over performance


Resource and process
control


Robust fault tolerance

MPI


No such abstraction


Rich message support


Support logical communication
topologies


Some realizations do not
interoperate across architectural
boundaries


Performance over flexibility


Primarily concerned with
messaging


More susceptible to faults

Each API has it’s unique strengths

Conclusion

PVM is better for:


Heterogeneous cluster,
resource and process
control


The size of cluster and
the time of program’s
execution are great


MPI is better for:


Supercomputers (
PVM is
not supported
)


Application for MPP




Max performance


Application needs rich
message support



Each API has it’s unique strengths

Acknowledgments


Scientific adviser


Assoc. Prof.

A.V.Komolkin

???

Thank you for your attention!