Graduate Computer Architecture I
Lecture 11: Distribute
Memory Multiprocessors
2
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Natural Extensions of Memory System
P
1
Switch
Main memory
P
n
(Interleaved)
(Interleaved)
First
-
level $
P
1
$
Inter
connection network
$
P
n
Mem
Mem
P
1
$
Inter
connection network
$
P
n
Mem
Mem
Shared Cache
Centralized Memory
Dance Hall, UMA
Distributed Memory (NUMA)
Scale
3
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Fundamental Issues
1.
Naming
2.
Synchronization
3.
Performance: Latency and Bandwidth
4
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Fundamental Issue #1: Naming
•
Naming
–
what data is shared
–
how it is addressed
–
what operations can access data
–
how processes refer to each other
•
Choice of naming affects
–
code produced by a compiler
•
via load where just remember address or keep track of
processor number and local virtual address for msg. passing
–
replication of data
•
via load in cache memory hierarchy or via SW replication and
consistency
5
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Fundamental Issue #1: Naming
•
Global
physical
address space
–
any processor can generate, address, and access it in
a single operation
–
memory can be anywhere: virtual addr. translation
handles it
•
Global
virtual
address space
–
if the address space of each process can be
configured to contain all shared data of the parallel
program
•
Segmented shared address space
–
locations are named <process number, address>
uniformly for all processes of the parallel program
6
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Fundamental Issue #2: Synchronization
•
Message passing
–
implicit coordination
–
transmission of data
–
arrival of data
•
Shared address
–
explicitly coordinate
–
write a flag
–
awaken a thread
–
interrupt a processor
7
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Parallel Architecture Framework
•
Programming Model
–
Multiprogramming
•
lots of independent jobs
•
no communication
–
Shared address space
•
communicate via memory
–
Message passing
•
send and receive messages
•
Communication Abstraction
–
Shared address space
•
load, store, atomic swap
–
Message passing
•
send, recieve library calls
–
Debate over this topic
•
ease of programming vs. scalability
8
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Scalable Machines
•
Design trade
-
offs for the machines
–
specialize vs commodity nodes
–
capability of node
-
to
-
network interface
–
supporting programming models
•
Scalability
–
avoids inherent design limits on resources
–
bandwidth increases with increase in resource
–
latency does not increase
–
cost increases slowly with increase in resource
9
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Bandwidth Scalability
•
Fundamentally limits bandwidth
–
Amount of wires
–
Bus vs. Network Switch
P
M
M
P
M
M
P
M
M
P
M
M
S
S
S
S
T
y
p
i
c
a
l
s
w
i
t
c
h
e
s
B
u
s
M
u
l
t
i
p
l
e
x
e
r
s
C
r
o
s
s
b
a
r
10
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Dancehall Multiprocessor Organization
S
c
a
l
a
b
l
e
n
e
t
w
o
r
k
P
$
S
w
i
t
c
h
M
P
$
P
$
P
$
M
M
S
w
i
t
c
h
S
w
i
t
c
h
11
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Generic Distributed System Organization
S
c
a
l
a
b
l
e
n
e
t
w
o
r
k
Comm
Assist
P
$
S
w
i
t
c
h
M
S
w
i
t
c
h
S
w
i
t
c
h
12
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Key Property of Distributed System
•
Large number of independent
communication paths between nodes
–
allow a large number of concurrent transactions
using different wires
•
Independent Initialization
•
No global arbitration
•
Effect of a transaction only visible to the
nodes involved
–
effects propagated through additional
transactions
13
-
CSE/ESE 560M
–
Graduate Computer Architecture I
CAD
Multipr
ogramming
Shar
ed
addr
ess
Message
passing
Data
parallel
Database
Scientific modeling
Parallel applications
Pr
ogramming models
Communication abstraction
User/system boundary
Compilation
or library
Operating systems support
Communication har
dwar
e
Physical communication medium
Har
dwar
e/Softwar
e boundary
Network Transactions
Programming Models Realized by Protocols
14
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Network Transaction
•
Interpretation of the message
–
Complexity of the message
•
Processing in the Comm. Assist
–
Processing power
P
M
CA
P
M
CA
°
°
°
Scalable Network
Node Architecture
Communication Assist
Message
Output Processing
–
checks
–
translation
–
formatting
–
scheduling
Input Processing
–
checks
–
translation
–
buffering
–
action
15
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Shared Address Space Abstraction
•
Fundamentally a two
-
way request/response protocol
–
writes have an acknowledgement
•
Issues
–
fixed or variable length (bulk) transfers
–
remote virtual or physical address
–
deadlock avoidance and input buffer full
–
Memory coherency and consistency
S
o
u
r
c
e
D
e
s
t
i
n
a
t
i
o
n
T
i
m
e
L
o
a
d
[
G
l
o
b
a
l
a
d
d
r
e
s
s
]
R
e
a
d
r
e
q
u
e
s
t
R
e
a
d
r
e
q
u
e
s
t
M
e
m
o
r
y
a
c
c
e
s
s
R
e
a
d
r
e
s
p
o
n
s
e
(
1
)
I
n
i
t
i
a
t
e
m
e
m
o
r
y
a
c
c
e
s
s
(
2
)
A
d
d
r
e
s
s
t
r
a
n
s
l
a
t
i
o
n
(
3
)
L
o
c
a
l
/
r
e
m
o
t
e
c
h
e
c
k
(
4
)
R
e
q
u
e
s
t
t
r
a
n
s
a
c
t
i
o
n
(
5
)
R
e
m
o
t
e
m
e
m
o
r
y
a
c
c
e
s
s
(
6
)
R
e
p
l
y
t
r
a
n
s
a
c
t
i
o
n
(
7
)
C
o
m
p
l
e
t
e
m
e
m
o
r
y
a
c
c
e
s
s
W
a
i
t
R
e
a
d
r
e
s
p
o
n
s
e
16
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Shared Physical Address Space
S
c
a
l
a
b
l
e
n
e
t
w
o
r
k
P
$
M
e
m
o
r
y
m
a
n
a
g
e
m
e
n
t
u
n
i
t
D
a
t
a
L
d
R
A
d
d
r
P
s
e
u
d
o
m
e
m
o
r
y
P
s
e
u
d
o
-
p
r
o
c
e
s
s
o
r
D
e
s
t
R
e
a
d
A
d
d
r
S
r
c
T
a
g
D
a
t
a
T
a
g
R
r
s
p
S
r
c
O
u
t
p
u
t
p
r
o
c
e
s
s
i
n
g
∙
M
e
m
a
c
c
e
s
s
∙
R
e
s
p
o
n
s
e
C
o
m
m
m
u
n
i
c
a
t
i
o
n
I
n
p
u
t
p
r
o
c
e
s
s
i
n
g
∙
P
a
r
s
e
∙
C
o
m
p
l
e
t
e
r
e
a
d
P
$
M
M
U
M
e
m
P
s
e
u
d
o
-
m
e
m
o
r
y
P
s
e
u
d
o
-
p
r
o
c
e
s
s
o
r
M
e
m
a
s
s
i
s
t
17
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Shared Address Abstraction
•
Source and destination data addresses are
specified by the source of the request
–
a degree of logical coupling and trust
•
No storage logically “outside the address space”
–
may employ temporary buffers for transport
•
Operations are fundamentally request response
•
Remote operation can be performed on remote
memory
–
logically does not require intervention of the remote
processor
18
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Message passing
•
Bulk transfers
•
Synchronous
–
Send completes after matching recv and
source data sent
–
Receive completes after data transfer complete
from matching send
•
Asynchronous
–
Send completes after send buffer may be
reused
19
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Synchronous Message Passing
•
Constrained programming model
•
Destination contention very limited
•
User/System boundary
S
o
u
r
c
e
D
e
s
t
i
n
a
t
i
o
n
T
i
m
e
S
e
n
d
P
d
e
s
t
,
l
o
c
a
l
V
A
,
l
e
n
S
e
n
d
-
r
d
y
r
e
q
T
a
g
c
h
e
c
k
(
1
)
I
n
i
t
i
a
t
e
s
e
n
d
(
2
)
A
d
d
r
e
s
s
t
r
a
n
s
l
a
t
i
o
n
o
n
P
s
r
c
(
4
)
S
e
n
d
-
r
e
a
d
y
r
e
q
u
e
st
(
6
)
R
e
p
l
y
t
r
a
n
s
a
c
t
i
o
n
W
a
i
t
R
e
c
v
P
s
r
c
,
l
o
c
a
l
V
A
,
l
en
R
e
c
v
-
r
d
y
r
e
p
l
y
D
a
t
a
-
x
f
e
r
r
e
q
(
5
)
R
e
m
o
t
e
c
h
e
c
k
f
o
r
p
o
s
t
e
d
r
e
c
e
i
v
e
(
a
s
s
u
m
e
s
u
c
c
e
s
s
)
(
7
)
B
u
l
k
d
a
t
a
t
r
a
n
s
f
e
r
S
o
u
r
c
e
V
A
D
e
s
t
V
A
o
r
I
D
(
3
)
L
o
c
a
l
/
r
e
m
o
t
e
c
h
e
c
k
20
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Asynch Message Passing: Optimistic
•
More powerful programming model
•
Wildcard receive
湯n
-
摥瑥牭t湩獴楣
•
Storage required within msg layer?
S
o
u
r
c
e
D
e
s
t
i
n
a
t
i
o
n
T
i
m
e
S
e
n
d
(
P
d
e
s
t
,
l
o
c
a
l
V
A
,
l
e
n
)
(
1
)
I
n
i
t
i
a
t
e
s
e
n
d
(
2
)
A
d
d
r
e
s
s
t
r
a
n
s
l
a
t
i
o
n
(
4
)
S
e
n
d
d
a
t
a
R
e
c
v
P
s
r
c
,
l
o
c
a
l
V
A
,
l
e
n
D
a
t
a
-
x
f
e
r
r
e
q
T
a
g
m
a
t
c
h
A
l
l
o
c
a
t
e
b
u
f
f
e
r
(
3
)
L
o
c
a
l
/
r
e
m
o
t
e
c
h
e
c
k
(
5
)
R
e
m
o
t
e
c
h
e
c
k
f
o
r
p
o
s
t
e
d
r
e
c
e
i
v
e
;
o
n
f
a
i
l
,
a
l
l
o
c
a
t
e
d
a
t
a
b
u
f
f
e
r
21
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Active Messages
•
User
-
level analog of network transaction
–
transfer data packet and invoke handler to extract it from the
network and integrate with on
-
going computation
•
Request/Reply
•
Event notification: interrupts, polling, events?
•
May also perform memory
-
to
-
memory transfer
Request
handler
handler
Reply
22
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Message Passing Abstraction
•
Source knows send data address, dest. knows
receive data address
–
after handshake they both know both
•
Arbitrary storage “outside the local address
spaces”
–
may post many sends before any receives
–
non
-
blocking asynchronous sends reduces the
requirement to an arbitrary number of descriptors
•
Fundamentally a 3
-
phase transaction
–
includes a request / response
–
can use optimistic 1
-
phase in limited “Safe” cases
23
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Data Parallel
•
Operations can be performed in parallel
–
each element of a large regular data structure, such as
an array
–
Data parallel programming languages lay out data to
processor
•
Processing Element
–
1 Control Processor broadcast to many PEs
–
When computers were large, could amortize the control
portion of many replicated PEs
–
Condition flag per PE so that can skip
–
Data distributed in each memory
•
Early 1980s VLSI
卉䵄S牥扩牴h
–
32 1
-
bit PEs + memory on a chip was the PE
24
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Data Parallel
•
Architecture Development
–
Vector processors have similar ISAs,
but no data placement restriction
–
SIMD led to Data Parallel Programming
languages
–
Single Program Multiple Data (SPMD) model
–
All processors execute identical program
•
Advanced VLSI Technology
–
Single chip FPUs
–
Fast µProcs (SIMD less attractive)
25
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Cache Coherent System
•
Invoking coherence protocol
–
state of the line is maintained in the cache
–
protocol is invoked if an “access fault” occurs
on the line
•
Actions to Maintain Coherence
–
Look at states of block in other caches
–
Locate the other copies
–
Communicate with those copies
26
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Scalable Cache Coherence
S
c
a
l
a
b
l
e
n
e
t
w
o
r
k
C
A
P
$
S
w
i
t
c
h
M
S
w
i
t
c
h
S
w
i
t
c
h
Realizing Program Models through
net transaction protocols
-
efficient node
-
to
-
net interface
-
interprets transactions
Caches naturally replicate data
-
coherence through bus snooping protocols
-
consistency
Scalable Networks
-
many simultaneous transactions
Scalable
distributed
memory
Need cache coherence protocols that scale!
-
no broadcast or single point of order
27
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Bus
-
based Coherence
•
All actions done as broadcast on bus
–
faulting processor sends out a “search”
–
others respond to the search probe and take necessary
action
•
Could do it in scalable network too
–
broadcast to all processors, and let them respond
•
Conceptually simple, but doesn’t scale with p
–
on bus, bus bandwidth doesn’t scale
–
on scalable network, every fault leads to at least p
network transactions
28
-
CSE/ESE 560M
–
Graduate Computer Architecture I
One Approach: Hierarchical Snooping
•
Extend snooping approach
–
hierarchy of broadcast media
–
processors are in the bus
-
or ring
-
based multiprocessors at the
leaves
–
parents and children connected by two
-
way snoopy interfaces
–
main memory may be centralized at root or distributed among
leaves
•
Actions handled similarly to bus, but not full broadcast
–
faulting processor sends out “search” bus transaction on its bus
–
propagates up and down hierarchy based on snoop results
•
Problems
–
high latency: multiple levels, and snoop/lookup at every level
–
bandwidth bottleneck at root
29
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Scalable Approach: Directories
•
Directory
–
Maintain cached block copies
–
Maintain memory block states
–
On a miss in own memory
•
Look up directory entry
•
Communicate only with the nodes with copies
–
Scalable networks
•
Communication through network transactions
•
Different ways to organize directory
30
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Basic Directory Transactions
P
A
M/D
C
P
A
M/D
C
P
A
M/D
C
Read request
to directory
Reply with
o
wner identity
Read req.
to o
wner
Data
Reply
Re
vision message
to directory
1.
2.
3.
4a.
4b
.
P
A
M/D
C
P
A
M/D
C
P
A
M/D
C
RdEx request
to directory
Reply with
sharers identity
In
v
al. req.
to sharer
1.
2.
P
A
M/D
C
In
v
al. req.
to sharer
In
v
al. ack
In
v
al. ack
3a.
3b
.
4a.
4b
.
Requestor
Node with
dirty
cop
y
Dir
ectory
node
for block
Requestor
Dir
ectory
node
Shar
er
Shar
er
(a) Read miss to a block in dirty state
(b)
Write miss to a block with tw
o sharers
31
-
CSE/ESE 560M
–
Graduate Computer Architecture I
E
S
I
P1
$
E
S
I
P2
$
D
S
U
M
Dir
ctrl
ld vA
-
> rd pA
Read pA
R/reply
R/req
P1: pA
S
S
Example Directory Protocol (1
st
Read)
32
-
CSE/ESE 560M
–
Graduate Computer Architecture I
E
S
I
P1
$
E
S
I
P2
$
D
S
U
M
Dir
ctrl
R/reply
R/req
P1: pA
ld vA
-
> rd pA
P2: pA
R/req
R/_
R/_
R/_
S
S
S
Example Directory Protocol (Read Share)
33
-
CSE/ESE 560M
–
Graduate Computer Architecture I
E
S
I
P1
$
E
S
I
P2
$
D
S
U
M
Dir
ctrl
st vA
-
> wr pA
R/reply
R/req
P1: pA
P2: pA
R/req
W/req E
R/_
R/_
R/_
Invalidate pA
Read_to_update pA
Inv ACK
RX/invalidate&reply
S
S
S
D
E
reply xD(pA)
Inv/_
Excl
Example Directory Protocol (Wr to shared)
I
34
-
CSE/ESE 560M
–
Graduate Computer Architecture I
A Popular Middle Ground
•
Two
-
level “hierarchy”
–
Coherence across nodes is directory
-
based
•
directory keeps track of nodes, not individual
processors
–
Coherence within nodes is snooping or
directory
•
orthogonal, but needs a good interface of
functionality
•
Examples
–
Convex Exemplar: directory
-
directory
–
Sequent, Data General, HAL: directory
-
snoopy
35
-
CSE/ESE 560M
–
Graduate Computer Architecture I
P
C
Snooping
B1
B2
P
C
P
C
B1
P
C
Main
Mem
Main
Mem
Adapter
Snooping
Adapter
P
C
B1
Bus (or Ring)
P
C
P
C
B1
P
C
Main
Mem
Main
Mem
Netw
ork
Assist
Assist
Netw
ork2
P
C
A
M/D
Netw
ork1
P
C
A
M/D
Directory adapter
P
C
A
M/D
Netw
ork1
P
C
A
M/D
Directory adapter
P
C
A
M/D
Netw
ork1
P
C
A
M/D
Dir/Snoop
y adapter
P
C
A
M/D
Netw
ork1
P
C
A
M/D
Dir/Snoop
y adapter
(a) Snooping
-
snooping
(b) Snooping
-
directory
Dir
.
Dir
.
(c) Directory
-
directory
(d) Directory
-
snooping
Two
-
level Hierarchies
36
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Memory Consistency
•
Memory Coherence
–
Consistent view of
the memory
–
Not ensure how
consistent
–
In what order of
execution
M
e
m
o
r
y
P
1
P
2
P
3
M
e
m
o
r
y
M
e
m
o
r
y
A
=
1
;
f
l
a
g
=
1
;
w
h
i
l
e
(
f
l
a
g
=
=
0
)
;
p
r
i
n
t
A
;
A
:
0
f
l
a
g
:
0
-
>
1
I
n
t
e
r
c
o
n
n
e
c
t
i
o
n
n
e
t
w
o
r
k
1
:
A
=
1
2
:
f
l
a
g
=
1
3
:
l
o
a
d
A
D
e
l
a
y
P
1
P
3
P
2
(
b
)
(
a
)
C
o
n
g
e
s
t
e
d
p
a
t
h
37
-
CSE/ESE 560M
–
Graduate Computer Architecture I
Memory Consistency
•
Relaxed Consistency
–
Allows Out
-
of
-
order Completion
–
Different Read and Write ordering models
–
Increase in Performance but possible errors
•
Current Systems
–
Relaxed Models
–
Expectation for synchronous programs
–
Use of standard synchronization libraries
- Αρχική
- Περιήγηση
- Όλες οι Κατηγορίες
- Internet και Εφαρμογές Web
- Αποθήκευση
- Ασφάλεια
- Βιοτεχνολογία
- Δίκτυα και Επικοινωνίες
- Διακομιστές
- Διαχείριση
- Διαχείριση Δεδομένων
- Επιχειρησιακές Εφαρμογές
- Ηλεκτρονική - Συσκευές
- Ημιαγωγοί
- Κινητά – Ασύρματες Τεχνολογίες
- Λογισμικό & κατασκευή λογ/κού
- Μηχανική
- Περιβαλλοντική Τεχνολογία
- Πετρελαϊκά και Εξόρυξη
- Πολεοδομικά Έργα
- Τεχνίτη Νοημοσύνη και Ρομποτική
- Τηλεπικοινωνίες
- Ψυχαγωγία
- Κοινότητα
- Μεταφόρτωση
Σχόλια 0
Συνδεθείτε για να κοινοποιήσετε σχόλιο