Georgia tech Routing Algorithms

VINetworking and Communications

Oct 6, 2011 (6 years and 1 month ago)

936 views

Georgia tech Routing Algorithms

1
Routing AlgorithmsRouting Algorithms-- II
© Sudhakar Yalamanchili, Georgia Institute of Technology (except as indicated)
Reading Assignment
Reading Assignment
• Sections 4.1-4.5

Alternative: relevant papers

Alternative: relevant papers
.

 References provided
ECE 8813a (2)
2
OverviewOverview
• Separate the routing decisions from
implementation
 Choice of path vs.
 Performance
o E.g., buffering, route computation implementation
• Routing Algorithm fixed by
 Routing function
 Selection function
P ima co ectness conce n
A single function for deterministic protocols
ECE 8813a (3)

P
r
ima
ry
co
rr
ectness conce
r
n
 Deadlock and livelock freedom
• Focus in key principles
 Not coverage of all algorithms designed to date
Virtual Channel Router
Virtual Channel Router
S
ECE 8813a (4)
D
Note
: Separation of physical and
virtual paths, e.g., shortest path
may traverse multiple virtual
networks
3
TaxonomyTaxonomy
Routing Algorithms
Unicast
multicast
Centralized
Source
Distributed
Multi
-
phase
# Destinations
Routing Decisions
Centralized
Source
Distributed
Multi
-
phase
Table Lookup Finite State Machine
Deterministic Adaptive
Progressive Backtracking
Routing Decisions
Implementation
Adaptivity
Progressiveness
ECE 8813a (5)
Profitable Misrouting
Complete Partial
Progressiveness
Minimality
Number of Paths
Classes of Routing AlgorithmsClasses of Routing Algorithms
• Deterministic and Oblivious Routing
 Deterministic protocols are oblivious
o Converse may not be true
 Variety of implementations
o Source Routing, finite state machines, interval routing,
table-lookup
• Partially Adaptive
• Fully, maximal, and true fully adaptive
 Fully: maximize alternative physical paths
ECE 8813a (6)
 Maximal: maximize all routing options
 True: no constraints on VC usage
• Non-minimal Routing
• Hybrid or multi-phase protocols
4
Deterministic Routing AlgorithmsDeterministic Routing Algorithms
Mesh
Binary Hypercube
ECE 8813a (7)
• Strictly increasing or decreasing order of
dimension
• Routing function always returns the same
output channel for each destination
Deterministic routing Deterministic routing
Tori
Tori
• Dimension order routing
• Create an acyclic channel
de
p
endenc
y

g
ra
p
h with the
n0
n1
c
0
c
1
c
3
p y g p
following routing function - c
0i
when j<i, c
1i
when j >i
n3
n2
c
2
c
3
n0
n1
c
10
c
00
c
10
c
11
c
01
c
03
ECE 8813a (8)
c
13
n3
n2
c
11
c
02
c
12
c
01
c
03
c
00
c
12
c
02
Deterministic routing
Deterministic routing
5
Physical RouterPhysical Router
Finite state
machine or Table-
look up
Distributed Routing
ECE 8813a (9)
Deterministic routing Deterministic routing
Virtual Channel Router
Virtual Channel Router
Finite state
machine or
Table-look up
Performance
constraints
Distributed Routing
ECE 8813a (10)
Remember, we can have virtual lanes
Deterministic routing Deterministic routing
6
Deterministic Routing Algorithms: Deterministic Routing Algorithms:
Implementation Issues Implementation Issues
• Relatively, the most inexpensive to implement
 Each node implements a routing function
o Shared or private logic across channels
 Absolute vs. relative addressing
• Header update
 Necessary for relative addressing
 Necessary to maintain uniformity of implementation
ECE 8813a (11)
• Implementation
 Table look-up vs. finite state machine
Deterministic routing
Deterministic routing
Source Routing
Source Routing
E
N
W
index
output port
• All routes are pre-computed at the source

却潲敤S慳⁡a獥煵敮捥s潦⁰潲琠慤摲敳獥猠慴a
ECE 8813a (12)

Stored as a sequence of port addresses at
intermediate routers
• Each intermediate router uses the header flit to
identify the output port
 Can be extended to use virtual channels
Oblivious routing
Oblivious routing
7
CharacteristicsCharacteristics
• Simple fast, route computation at intermediate
routers
 Header update
• Oblivious routing
 The source-destination pair does not need to identify
a unique path  not deterministic
• Complexity of computing paths
 Ensuring deadlock freedom?
We will come
b k t thi
E C E 8 8 1 3 a ( 1 3 )
• Extensions to virtual channels
Oblivious routing
Oblivious routing
b
ac
k t
o
thi
s
Filling in the Routing Tables
Filling in the Routing Tables
N
4
E
1
S
0
index
index at next
node
• How do you find deadlock free paths?
 Non-trivial problem

Short
-
cuts
S
1
ECE 8813a (14)
Short
cuts
 Obtain a channel dependency graph for a specific
algorithm
 Find paths in channel dependency graphs!
 End with virtual channel assignment
Oblivious routing
Oblivious routing
From M. Kinsy, et.al., ”Application Aware, Deadlock Free Oblivious
Routing,” Proceedings of ISCA 2009
8
ApplicationApplication--Specific Oblivious RoutingSpecific Oblivious Routing
Channel Dependence Graph
Flow Graph
Two different Channel Dependency Graphs
ECE 8813a (15)
From M. Kinsy, et.al., ”Application Aware, Deadlock Free Oblivious
Routing,” Proceedings of ISCA 2009
Oblivious routing Oblivious routing
Interval Routing
Interval Routing
0
4
8
12
1
5
9
13
Channel Node Interval
X+ 6 8-15
X
- 6 0-3
• Each output link corresponds to an interval of nodes

啮楯n of⁩ tervals at⁡ode 楳⁴he set of⁡汬ldest楮at楯nodes
2
6


3
7


夫 6 4ⴵ
夭 6 7
ECE 8813a (16)

Union of intervals at a node is the set of all destination nodes
 Must be able to distinguish invalid intervals (at the edges of a
mesh)
• Use overlapping intervals for fault tolerance
Oblivious routing Oblivious routing
J.V. Leeuwen and R. B. Tan, “Interval Routing,” The Computer Journal,
vol. 30, no.4, 1987
9
Partially Adaptive Routing AlgorithmsPartially Adaptive Routing Algorithms
• Trade-off between hardware resources and
adaptivity
 Maximize adaptivity for given resources
 Minimize resources required for a given level of
adaptivity
• Typically exploit regular topologies
ECE 8813a (17)
Partially Adaptive Partially Adaptive
Routing Routing
A. A. Chien and J. H. Kim, “Planar Adaptive Networks: Low Cost
Adaptive Networks for Multiprocessors, ISCA 1992
Planar Adaptive RoutingPlanar Adaptive Routing
A0
A1
A0
A1
A2
• Packets are routed adaptively in a series of two
dimensional planes
 Order of planes (dimensions) is arbitrary
Fully adaptive
Planar adaptive Planar adaptive
ECE 8813a (18)
• Routin
g
in two dimension uses two virtual
networks
 Increasing and decreasing networks
Partially Adaptive Routing
Partially Adaptive Routing
10
Adaptive Routing in Two Dimensions Adaptive Routing in Two Dimensions
0
1
2
3
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
4
5
6
7
8
9
10
11
12
13
14
15
ECE 8813a (19)
Increasing Network
Decreasing Network
D
i+1
D
i
Partially Adaptive Routing Partially Adaptive Routing
PAR in Multidimensional Networks
PAR in Multidimensional Networks
Dimension i+2
A0
Dimension i
Dimension i+1
ECE 8813a (20)
• Routing is fully
adaptive in a plane
• When can you skip a
plane?
A1
Partially Adaptive Routing Partially Adaptive Routing
11
PAR PropertiesPAR Properties
• Each plane is comprised of the following
channels
A
i
= d
i,2
+ d
i+1,0
+ d
i+1,1
• Three virtual channels/link in meshes and six
virtual channels/link in Tori
ECE 8813a (21)
Partially Adaptive Routing Partially Adaptive Routing
The Turn Model
The Turn Model
• What is a turn?
 From one dimension to another : 90 degree turn
 To another virtual channel in the same direction: 0
degree turn

呯⁴桥T牥癥牳攠摩牥捴楯渺rㄸ〠摥杲敥d瑵牮
abstract cycles XY routing west-first routing
ECE 8813a (22)

To the reverse direction: 180 degree turn
• Turns combine to form cycles
• Goal: prohibit the least number of turns to
break all possible cycles
Partially Adaptive Partially Adaptive
Routing Routing
C. J. Glass and L. Ni, “The Turn Model for Adaptive Routing,” ISCA 1992
12
Turn ConstraintsTurn Constraints
• Choice of prohibited turns is not arbitrary
equivalent
ECE 8813a (23)
• Alternative designs
 Three combinations unique (within symmetry)
o Three algorithms: west-first, north-last, negative-first
Partially Adaptive Routing
Partially Adaptive Routing
West First Routing
West First Routing
deterministic region
fully adaptive region
ECE 8813a (24)
• Fully adaptive to the east, deterministic to the
west (for non-minimal routing)
• Non-minimal is partially adaptive to the west
Partially Adaptive Routing Partially Adaptive Routing
13
Generalization of the Turn ModelGeneralization of the Turn Model
Application to Binary hypercubes
•Base is e-cube routing
•Identify up channels and down
channels
• Identify channel classes and prohibit turns
between them
Cycles are infeasible within a channel class
•Adaptively route though one class
and then the other
•Turns prohibited from one class
to the other
ECE 8813a (25)

Cycles are infeasible within a channel class
• Transitions between channel classes are acyclic
• Partially adaptive: number of shortest paths
are reduced
Partially Adaptive Routing Partially Adaptive Routing
P
P--Cube RoutingCube Routing
0010
1111
1101
up channel
• For a given source destination pair compute
the up channel set and down channel set
• Restrict turns from one set to the other
0000
1101
down channel
ECE 8813a (26)
• Adaptively route within a set
• The escape path corresponds to some fixed
order which remains a subset of channels
14
Fully Adaptive RoutingFully Adaptive Routing
• Using all available paths
 Minimal vs. non-minimal
• Distinguish between buffer-based schemes vs.
channel-based schemes
 Typically the former can be translated to the latter
ECE 8813a (27)
Fully Adaptive Routing
Fully Adaptive Routing
Structured Buffer Pools
Structured Buffer Pools
buffer 1
buffer D-1
• Positive hop algorithm
buffer 0
ECE 8813a (28)
 D+1 buffers at each node (D = diameter)
 Nodes request/use buffers in strictly increasing order
 Minimal path algorithm, valid for any topology
 Large buffer requirements: O(Diameter)
Fully Adaptive
Fully Adaptive
Routing Routing
I. S. Gopal, “Prevention of Store and Forward Deadlock in Computer
Networks,” IEEE Transactions on Communication, December 1985.
15
Structured Buffer Pools: ExtensionStructured Buffer Pools: Extension
set 0
subset S-1subset 1subset 0
• Negative hop algorithm
 Partition nodes into non-adjacent subsets
 Order subsets
set 1
ECE 8813a (29)
 Down transitions request higher numbered buffer,
else same numbered buffer
 Number of buffers required in each node is given by
+ 1







S
SD )1(
Fully Adaptive Routing
Fully Adaptive Routing
Extensions to Wormhole Switching
Extensions to Wormhole Switching
Bu
ffers
Physical
it B
uffers
Node Node
b
1
b
k
c
i
c
i,1
c
i,k
replace central buffers with
equivalent number of
virtual channels across
each physical channel
Acyclic central buffer
dependencies are
transformed into
acyclic virtual channel
dependencies
Bu
Physical
Channel
Fl
it B
Virtu
al
Channe
ls
b
m
c
i,m
Store-and-Forward Switching Wormhole Switching
• Basic version produces unbalanced use of virtual channels
 distance rarely equal to diameter
• Extension: bonus cards.
ECE 8813a (30)
 Number is equal to unused hops: diameter -#req
 Use this number to increase the number of choices of virtual
channels at any node
 Not the same as adaptivity
Fully Adaptive Routing
Fully Adaptive Routing
16
Virtual NetworksVirtual Networks
• Establish multiple virtual networks
 Each network works for a specific destination set
 Routing functions in a network are acyclic, but are
typically not connected
• Establish routing constraints between virtual
networks

Simple customized protocols
ECE 8813a (31)

Simple customized protocols
 Expensive in terms of virtual channels
Fully Adaptive Routing
Fully Adaptive Routing
Virtual Networks in a Mesh
Virtual Networks in a Mesh
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
Y
8
9
10
11
12
13
14
15
8
9
10
11
12
13
14
15
X
X+Y+ X-Y+
• Each virtual network is constructed to have acyclic
h l d d i
ECE 8813a (32)
c
h
anne
l d
epen
d
enc
i
es
 Routing function in a virtual network is not connected
• Packets are injected into the appropriate virtual network
• Fully adaptive, no transitions between networks
• 2
n
virtual networks with (n.2
n
) virtual channels/node
Fully Adaptive Routing Fully Adaptive Routing
17
ReferencesReferences
• C.R. Jesshope, P.R. Miller, and J.T. Yantchev,
“High performance communications in
processor networks,'‘ Proceedings of the 16th
International Symposium on Computer
Architecture, pp.150-157, May-June 1989
• D.H. Linder and J.C. Harden, ``An adaptive
and fault tolerant wormhole routing strategy
for k-ar
y
n-cubes
,
'‘ IEEE
T
ransactions on
ECE 8813a (33)
y
,
Computers, vol. C-40, no. 1, pp.2-12, January
1991
Virtual Networks in a MeshVirtual Networks in a Mesh
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
Y
8
9
10
11
12
13
14
15
8
9
10
11
12
13
14
15
X
X+Y+ X-Y+
• Extensions to 3D
ECE 8813a (34)
• Extensions to tori and irregular topologies
• Couple with source routing?
• Extensions to sub-topologies
• Limit adaptivity to a subset of nodes
18
OptimizeOptimize
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
• Reduce the number of networks by 50% with one
dditi l h l i di i 0
8
9
10
11
12
13
14
15
8
9
10
11
12
13
14
15
Virtual network 0 Virtual network 1
ECE 8813a (35)
a
dditi
ona
l
c
h
anne
l i
n
di
mens
i
on
0
• Total number of channels/node
 Number of channels in each network + extra channel in
dimensions 0 (in networks
Fully Adaptive Routing Fully Adaptive Routing
111
2)1(22


nnn
nn
1
2
n
Optimize
Optimize
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
• Extensions to k-ary n-cubes by introducing levels in each
i t l t k
8
9
10
11
12
13
14
15
8
9
10
11
12
13
14
15
Virtual network 0 Virtual network 1
ECE 8813a (36)
v
i
r
t
ua
l
ne
t
wor
k
 One level for each wrap-around channel
 At most n levels are traversed (in addition to the original
level)
 (n+1).2
(n-1)
virtual channels/physical channel for
dimensions >0
Fully Adaptive Routing Fully Adaptive Routing
19
Extensions to ToriExtensions to Tori
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
L = 2
• Addition of layers
(virtual networks)
for each wrap-
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
L = 1
around
connection
• Number of layers
increases by
#dimensions
ECE 8813a (37)
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
L = 0
Virtual Network 0 Virtual Network 1
Fully Adaptive Routing Fully Adaptive Routing
Using Dynamic Message Dependencies
Using Dynamic Message Dependencies
• Two virtual channel classes across each
physical link
 Adaptive channels & deterministic channels
• Fully adaptive use of adaptive channels
 Keep track of #dimension reversals for each message
o Moving from a dimension p to a lower dimension q
 Label each channel with the DR# of the message
 Messages cannot block on a channel with lower DR#
ECE 8813a (38)
o If no channel available, permanent transition to the
deterministic channel
 Dependencies between messages are acyclic
Fully Adaptive Routing
Fully Adaptive Routing
W.J. Dally and H. Aoki, “Deadlock-free adaptive routing in multicomputer networks using
virtual channels,” IEEE Transactions on Parallel and Distributed Systems,vol. 4, no. 4, pp.
466-475, April 1993.
20
Summary of Design TechniquesSummary of Design Techniques
• Ordered use of topological features
 Dimensions and paths
Ordered use of resources

Ordered use of resources
 Buffers/channels, and networks
• Order the message population
 Each message is uniquely identified by some
attribute, e.g., number of wrap-around channel
crossings
d bl k b d l
ECE 8813a (39)
 Or
d
er
bl
oc
k
ing
b
ase
d
on message popu
l
ation
membership
DesignDesign
Design Methodology
Design Methodology
• Start with a network, set of channels C
1
, and routing
function R
1
 R
1
is connected and deadlock free and may be
deterministic/adaptive minimal/non
-
minimal
deterministic/adaptive
,
minimal/non
-
minimal
• Split each physical channel into a set of additional virtual
channels and define the new routing function
 Set of channels includes escape channels and adaptive
channels

卥Se捴co渠晵湣瑩潮⁣慮o扥b摥晩湥搠in慮礠睡祳
))((),(),(
11
CCCyxRyxR
xy




Set of channels from
node x (local node) to
node y)
ECE 8813a (40)

Selection function can be defined in many ways
• For wormhole switching, verify that the extended channel
dependency graph is acyclic: likely if R is restricted to
minimal paths
DesignDesign
21
Example Binary HypercubeExample Binary Hypercube
n2 n3
b37b26
b32b23
• Start with
dimension order e-
cube algorithm
n6
n4
n7
n5
b54
b76
b64
b46
b
20
b
02
b45
b67
b75
b57
b1
3
b3
1
• Add additional
channels for
adaptive routing
6 7
32
ECE 8813a (41)
n0 n1
b15
b54
b04
b45
b01 b10
0
1
54
Design
Design
Maximally Adaptive Routing
Maximally Adaptive Routing
• Establish a relationship between routing
freedom and resources
 Maximize adaptivity for fix resources
 Minimize resources for target adaptivity
• Relationship between adaptivity and
performance
 Not obvious
 Unbalanced use of physical or virtual channel
ECE 8813a (42)
resources
Maximally Adaptive Routing
Maximally Adaptive Routing
22
ReferencesReferences
• C.J. Glass and L.M. Ni, “Maximally fully
adaptive routing in 2D meshes,” Proceedings of
the International Conference on Parallel
Processin
g
, Au
g
ust 1992.
• R. Cypher and L. Gravano, “Storage-efficient,
deadlock-free packet routing algorithms for
torus networks,” IEEE Transactions on
Computers, vol. 43, no. 12, pp.1376-1385,
December 1994
ECE 8813a (43)
• L. Schwiebert and D.N. Jayasimha, “Optimal
fully adaptive wormSupercomputing'93hole
routing for meshes,”, pp.782-791, November
1993
In 2D Meshes: Double YIn 2D Meshes: Double Y
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
Y1Y2
6
Y2
8
9
10
11
12
13
14
15
8
9
10
11
12
13
14
15
increasing network decreasing network
6
Y1
ECE 8813a (44)
Maximally Adaptive Routing
Maximally Adaptive Routing
X+
X- Y1-
Y1+
Y1+
Y1-
X-
X+
Y2+
Y2-
X-X+
X-
Y2+
X+Y2-
23
In 2D Meshes: MadIn 2D Meshes: Mad--YY
X+
X- Y1-
Y1+
Y1+
Y1-
X-
X+
Y2+
Y2-
X-X+
X-
Y2+
X+Y2-
permitted
6
Y2
Y1
• Permit turns
 From the Y1 channels to the X+ channels
 From X- channels to Y2 channels
No coupling
between Y2 and Y1
ECE 8813a (45)
• Remove unnecessary turn restrictions
• Still overly restrictive!
Maximally Adaptive Routing Maximally Adaptive Routing
In 2D Meshes: Opt Y
In 2D Meshes: Opt Y
X+
X- Y1-
Y1+ Y1+
Y1-
X-
X+
Y2+
Y2-
X-X+
X-
Y2+
X+Y2-
Y2+ Y1+
Y1+ Y2+
Y2- Y1-
Y1- Y2-
• Further reduce the number of restrictions
 Only restrict turns from Y1 to X-
 Turns from X- to Y1 and 0-degree turns in Y only
when X offset is 0 or positive
• Extensions to multidimensional meshes
B i id
f ll d ti ti i t f
ECE 8813a (46)

B
as
i
c
id
ea:
f
u
ll
y

a
d
ap
ti
ve

rou
ti
n
g i
n

one

se
t
o
f
channels, and dimension order in the other set
until specified lower dimension traversals are
complete
Maximally Adaptive Routing Maximally Adaptive Routing
24
Routing with Minimum Buffer Routing with Minimum Buffer
RequirementsRequirements
• Key Idea:
 Organize packet traffic into disjoint groups that use
separate buffers in each node
 Place acyclic routing restrictions in buffer usage
• Based on node orderings
ECE 8813a (47)
Maximally Adaptive Routing
Maximally Adaptive Routing
Node Labeling for 2D Torus
Node Labeling for 2D Torus
0
1
2
3
4
5
6
7
15
14
13
12
11
10
9
8
8
9
10
11
12
13
14
15
7
6
5
4
3
2
1
0
15
14
12
13
11
10
8
9
0
1
3
2
4
5
7
6
right increasing node ordering
left increasing node ordering
ECE 8813a (48)
3
2
0
1
7
6
4
5
12
13
15
14
8
9
11
10
outside increasing node orderinginside increasing node ordering
Maximally Adaptive Routing
Maximally Adaptive Routing
25
Algorithm 1Algorithm 1
• Algorithm
 Packet moves from the injection queue to the A
queue
 Stay in the A queues as long as we can
can
move to right
along at least one dimension along a minimal path
 Transition to the B queues under same rule for left
traversals
 Transition to the C queue and remain there until
packet is delivered
ECE 8813a (49)
• Note the de-coupling of node labeling from
buffer labeling
Maximally Adaptive Routing Maximally Adaptive Routing
Application
Application
0
1
• Orderings analogous
to virtual planes
 Note the orderings are
2
4
5
6
3
acyclic
• Extensions to edge
buffers
 Check Algorithm 2
ECE 8813a (50)
7
76543210
Maximally Adaptive Routing
Maximally Adaptive Routing
26
True Fully Adaptive RoutingTrue Fully Adaptive Routing
• Adaptivity extends across physical and virtual
channels
• Deadlock recovery vs. deadlock avoidance
ECE 8813a (51)
Maximally Adaptive Routing Maximally Adaptive Routing
Next
Next
• Non-minimal routing

Topology agnostic routing

Topology agnostic routing
• Extensions and application of similar principles
ECE 8813a (52)