# An Introduction to Parallel Computing

Λογισμικό & κατασκευή λογ/κού

1 Δεκ 2013 (πριν από 4 χρόνια και 5 μήνες)

108 εμφανίσεις

An Introduction to Parallel Computing

Dr. David Cronk

Innovative Computing Lab

University of Tennessee

Distribution A: Approved for public release; distribution is unlimited.

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

2

Outline

Parallel Architectures

Parallel Processing

What is parallel processing?

An example of parallel processing

Why use parallel processing?

Parallel programming

Programming models

Message passing issues

Data distribution

Flow control

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

3

Shared Memory Architectures

CPU

CPU

CPU

CPU

CPU

Main Memory

bus

Symmetric multiprocessors (SMPs)

Access time is uniform

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

4

Shared Memory Architectures

Non
-
Uniform Memory Access (NUMA)

Main Memory

bus

CPU

CPU

CPU

CPU

CPU

Main Memory

bus

CPU

CPU

CPU

CPU

CPU

Network

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

5

Distributed memory Architectures

M

M

M

M

M

M

M

M

P

P

P

P

P

P

P

P

Network

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

6

Networks

Grid

processors are connected to 4 neighbors

Cylinder

A closed grid

Torus

A closed cylinder

Hypercube

Each processor is connected to
2^n other processors, where n is the degree
of the hypercube

Fully Connected

Every processor is directly
connected to every other processor

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

7

Parallel Processing

What is parallel processing?

Using multiple processors to solve a single
problem

The problem consists of a number of independent

Each processor or groups of processors can perform

Data parallelism

The problem consists of dependent tasks

Each processor works on a different part of data

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

8

Parallel Processing

dx
x
1
0
2
)
1
(
4
We can approximate the integral as a sum of rectangles

N
i
x
0
i
)
F(x
David Cronk

Distribution A: Approved for public release; distribution is unlimited.

9

Parallel Processing

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

10

Parallel Processing

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

11

Parallel Processing

Why parallel processing?

Faster time to completion

Computation can be performed faster with
more processors

Able to run larger jobs or at a higher
resolution

Larger jobs can complete in a reasonable
amount of time on multiple processors

Data for larger jobs can fit in memory when

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

12

Parallel Programming

Outline

Programming models

Message passing issues

Data distribution

Flow control

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

13

Parallel Programming

Programming models

Shared memory

Distributed memory (message passing)

Data is shared via explicit message passing

Combination shared/distributed

data while data is shared between groups via
explicit message passing

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

14

Message Passing

Message passing is the most common method
for programming for distributed memory

With message passing, there is an explicit

In message passing systems, different
processes are identified by unique identifiers

Simplify this to each having a unique numerical
identifier

Senders send data to a specific process based on this
identifier

on this identifier

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

15

Parallel Programming

Message Passing Issues

Data Distribution

Latency (message start up time)

»
Few large messages is better than many small

Memory movement

Less idle time waiting for data or synchronizing

Each process should do about the same work

Flow Control

Minimize waiting

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

16

Data Distribution

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

17

Data Distribution

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

18

Flow Control

0 1 2 3 4 5 ………

Send to 1

Recv from 0

Send to 2

Recv from 1

Send to 3

Recv from 2

Send to 4

Send to 1

Send to 2

Recv from 0

Send to 3

Recv from 1

Send to 4

Recv from 2

David Cronk

Distribution A: Approved for public release; distribution is unlimited.

19

through support provided by DoD HPCMP

PET activities through Mississippi State

University (MSU) under contract

No. N62306
-
01
-
D
-
7110
.