Message Passing Interface vs. Multi-threading - Computing and ...

errorhandleΛογισμικό & κατασκευή λογ/κού

18 Νοε 2013 (πριν από 3 χρόνια και 8 μήνες)

170 εμφανίσεις

Presented to
George Seweryniak
Mathematical, Information, and Computational
Sciences
Jonathan C. Rann
Winston-Salem State University
Research Alliance in Mathematics and Science
Oak Ridge, Tennessee
August 9, 2006
Message Passing Interface vs. Multi-threading:
Which Parallelization Technique is More
Efficient?
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
OUTLINE
·
Motivation
·
Research goals
·
Sequential computing

Sequential architecture
·
Parallel computing

Parallel Architecture

Parallelization
·
Results
·
Research

Current

Future
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
·
To help create optimized biological applications which
utilizes the maximum efficiency of computer hardware
·
Implemented through


Message Passing Interface (MPI)

Multithreading
MOTIVATION
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
·
Typical application times

Web-page loading: 1-10 s

Unzip a large file: 10-100s

Biological applications: >1,000 hours
·
Parallel programming reduces time-to-solutions


Combined power of 100 CPUs will solve the problem in only10 hours!


Reduces the time required for biological research by speeding the
calculations


Modern supercomputer have >10,000 CPUs
MOTIVATION: Biological Applications
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
·
Modeling of biological molecules: DNA and proteins

Calculations require > 1,000 hours

Parallel programming is a technique to reduce time to solve the calculations

Parallel programming:
divide-and-conquer
approach
MOTIVATION: Biological Applications
CPU 1
CPU 3
CPU 2
CPU 4
Each processor does a part of the calculation
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
RESEARCH GOALS
·
Analyze parallel programming techniques
·
Create a parallelized program using both Message
Passing Interface and Multi-threading techniques
·
Compare the efficiency for created program
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
SEQUENTIAL COMPUTING
·
Traditional computing model

One processor used to execute each instruction

Each instruction executed one at a time, in sequence
Processor
Process
Memory
Instructions
Processor
1
2
3
4
TIME = 0
TIME = 1
TIME = 2
TIME = 3
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
SEQUENTIAL ARCHITECTURE
·
Single Instruction, Single Data
E.g. Each worker in a factory
making one car, thus creating a
more time consuming process
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
PARALLEL COMPUTING
·
Simultaneous use of multiple processors to
complete a specific task
Instructions
Processors
1
2
3
4
TIME

= 0
TIME = 0
TIME = 0
TIME = 0
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
PARALLEL ARCHITECTURES…

Multiple Instructions, Single Data (MISD)

Single Instruction, Multiple Data (SIMD)

Multiple Instruction, Multiple Data (MIMD)
E.g. Multiple workers
creating a single car
on an assembly line
E.g. Single worker executes
the same task on multiple
cars as they come down the
assembly line
E.g. Multiple workers execute
different tasks on different
cars
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
PARALLELIZATION
·
Involves…

Segmenting data
·
Master process breaks a task
down into smaller parts that can be
computed independently

Sending data
·
Slave process assigns, then sends
data segments to different processors

Compute
·
Upon receiving the tasks, the processor
computes them, then sends them back to
master process

Combine
·
Computed data now brought back
together; task is complete
Proc
Proc
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
MESSAGE PASSING
·
Message Passing Interface (MPI)

Industry standard for parallel programming

Used in writing C, C++, and Fortran

Enables communication between multiple processors through
specific commands
To: master
processor
slave
To: slave
processor
MASTER
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
MESSAGE PASSING (CONT'D)
·
Basic MPI commands:

MPI_Init()
·
Initializes communicator


MPI_Comm_size()
·
Determines total number of processors available in
communicator

MPI_Comm_rank()
·
Assigns each processor a number or position in
communicator

MPI_Send()
·
Sends data from one processor to another


MPI_Recv()
·
Receives data from another processor

MPI_Finalize()
·
Shut down the communicator
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
MULTI-THREADING
Technique used to split a program into two or
more simultaneously running tasks

Suitable for processors with multiple cores (ex.
Intel's Dual core processor) or even single core
processor

A single processor switches between different
threads when performed sequentially

Multiple threads running simultaneously on
different processors when performed in
parallel
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
RESEARCH
·
SAMPLE APPLICATION: Merge sort algorithm
sort(List[], left, right)
begin
if left < right then
mid = (left + right) / 2
sort(list[], left, mid)
sort(list[], mid + 1, right)
merge(list[], left, mid, right)
endif
End
9 3 4 220 1 3 10 5 8 7 2
9 3 4 220 1 3 10 5 8 7 2
9 3 4 220 1
3 10 5 8 7 2
1 3 4 9 220
2 3 5 7 8 10
1 2 3 3 4 5 7 8 9 10 220
Divide the data set in half
Sort each half
Merge the halves to
obtain sorted list
9 3 4 220 1 3 10 5 8 7 2
9 3 4 220 1 3 10 5 8 7 2
9 3 4 220 1
3 10 5 8 7 2
1 3 4 9 220
2 3 5 7 8 10
1 2 3 3 4 5 7 8 9 10 220
Divide the data set in half
Sort each half
Merge the halves to
obtain sorted list
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
RESEARCH (Contd.)
·
Create MPI implementing code that performs
merge sort
main(MPI
args
)
begin // master process
initialize MPI environment
get task identification and info (rank, proc)
Send MPI to sort
initialize sort
Receive MPI from sort
End MPI environment
end
sort(List
[], left, right)
begin // slave processes
if left < right then
Receive MPI
Mediate jobs among processors
mid = (left +
rigt
) / 2
sort(list
[], left, mid)
sort(list
[], mid + 1, right)
merge(list
[], left, mid, right)
Send sorted data to main
endif
end
int MPI_Init (…
.)
int MPI_Finalize ( )
int MPI_Comm_size(..)
int MPI_Comm_rank(..)
int MPI_Send (….)
int MPI_Recv (….)
int MPI_Recv (….)
int MPI_Send (….)
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
RESULTS
Sequential Mergesort
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
250000
500000
1000000
1500000
2000000
Number of Elements
T
i
m
e
(seco
n
d
s)
1 proc
RESULT
: Time required to compute a solution with the sequential
implementation of the merge sort program
increases
with the size of an array
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
RESULTS (Contd.)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Time (second
s)
250000
500000
1000000
1500000
2000000
Number of Elements
MPI Implementation of Mergesort
1 proc
2 procs
4 procs
8 proces
RESULT
: As the number of processors used to compute the solution with the
MPI implementation of the merge sort increases, the computation time
decreases

O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
·
Exploration of Multithreading

Better understanding of process synchronization
·
Resource sharing
·
Deadlock

Efficiency comparisons with MPI
CURRENT RESEARCH
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
FUTURE RESEARCH
·
Complete multi-threaded implementation
of merge sort
·
Characterize the efficiency of applications
using alternate parallel programming
methods
·
Create optimized applications based on
research
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
Acknowledgments
The Research Alliance in Math and Science program is
sponsored by the Mathematical, Information, and
Computational Sciences Division, Office of Advanced
Scientific Computing Research, U.S. Department of
Energy.
The work was performed at the Oak Ridge National
Laboratory, which is managed by UT-Battelle, LLC under
Contract

No. De-AC05-00OR22725. This work has been
authored by a contractor of the U.S. Government,
accordingly, the U.S. Government retains a non-exclusive,
royalty-free license to publish or reproduce the published
form of this contribution, or allow others to do so, for U.S
Government purposes.
O
AK
R
IDGE
N
ATIONAL
L
ABORATORY
U. S. D
EPARTMENT

OF
E
NERGY
Questions?