How The Port Density Of A Data Center LAN Switch Impacts

hellhollowreadingΔίκτυα και Επικοινωνίες

26 Οκτ 2013 (πριν από 3 χρόνια και 9 μήνες)

60 εμφανίσεις








How the Port Density of a Data Center
LAN Switch Impacts Scalability and
Total Cost of Ownership













June 4, 2012


2

Introduction

As data centers are forced to accommodate rapidly
growing volumes of information, their capacities to
process, s
tore, and transfer data are having difficulty
keeping pace. A Gartner survey
1

in November 2010
found that 47% of representatives from 1,004 large
enterprises from eight countries ranked data growth as
one of their top three challenges. This was followed by

system performance and scalability at 37%, and network
congestion and connectivity architecture at 36%.

Growing demand based on data proliferation, new
applications, and federated applications are
exhausting
the capacities of many data centers. According
to the
Uptime Institute’s 2011 Data Center Industry Survey

2
,
more than a third (36%) of data center facilities will run
out of space, power, or cooling between 2011 and 2012.
These findings were based on a survey of 525 data
center executives. Of the resp
ondents reaching full
capacity, 40% plan to build a new data center, while
62% plan to address the problem via server
consolidation. Server virtualization is the primary
mechanism for implementing server consolidation, but
server virtualization introduces

new sources of server
-
to
-
server traffic that pose challenges for the scalability
of the data center network. Server virtualization also
requires that the bandwidth of each physical server’s
network connection scale in proportion to the increased
demands f
or network I/O from multiple virtual
machines sharing the same physical server port(s).

As IT managers re
-
architect or upgrade their data
centers their focus will be on assuring the ability to
scale computing performance and networking capacity
within the

facility’s constraints of space, power, and
cooling capacities.

The goal of this white paper is help data center owners
and designers quantify the value of 40 Gigabit Ethernet
switch port density and scalability as part of a Total
Cost of Ownership (TCO)
approach to evaluating
alternative data center network designs and



1

http://www.informationweek.com/news/hardware/data_centers/229600034

2

http://gigaom2.files.wordpress.com/2011/06/inaugural
-
uptime
-
institute
-


annual
-
data
-
center
-
survey
-
fact
-
sheet.pdf

D
efinition of Symbols

P
: The number of 40

GbE ports per
aggregation switch


M
: The effective over
-
subscription

R
atio


S
:
The number of aggregation switches

Ccore
: The number of 40 GbE ports per
aggregation switch that are used to
connect to the LAN


core
Cagg
:
The number of 40 GbE ports
per aggreg
ation switch used to connect
to other aggregation switches


Cacc
:
The number of connections
between TOR switches


P


Ccore


Cagg
:
The number of 40
GbE ports per aggregation switch
available for connections to the access
layer


4 x m x (P
-
Ccore
-
Cagg)
:
The

number
of 10 GbE access layer ports that are
available for server connection per
aggregation


4 x S x m x (P
-
Ccore
-
Cagg)
: For two
tier LAN design with multiple
aggregation switches, the number of
available server ports



3

implementations with different brands of switching products. This white paper is part of an
on
-
going series of documents and webinars from Extreme Networks that focus on quantifying
the val
ue of the key attributes of a data center LAN switch.

The Impact of Server Virtualization on Data Center LANs

The widespread adoption of server virtualization within the data center is introducing new
traffic patterns that must be accommodated by the data
center network. In addition to the
traditional client/server traffic flows (“north
-
south” traffic), virtual machine migrations and
access to networked storage give rise to a significant amount of server
-
to
-
server (“east
-
west”)
traffic within PODs and even
across the entire data center. Virtual machine migrations among
physical servers are being used to support a number of critical data center functions, including
load balancing, expanding capacity, disaster recovery, and ensuring high availability. Some of
the impact that server virtualization has had on the data center network includes:



Flatter networks

Virtual machine migrations are typically performed within the boundaries of a VLAN.
Extending VLANs across wider segments of the data center requires the el
imination or
circumvention of Layer 3 boundaries between server PODs. Flatter networks can help
lower costs by reducing configuration complexity and by reducing the number of
devices that must be managed. Another benefit of flatter data center networks is
reduced end
-
to
-
end latency due to fewer hops between source and destination nodes.



Higher Bandwidth Server Connectivity

Servers with multi
-
core processors can support a large and growing number of virtual
machines, increasing the demand for network I/O cap
acity. The trend to multi
-
core
and multi
-
virtual machine (VM) servers is accelerating the need to transition server
connectivity from GbE to 10 GbE. Adding to that pressure is the fact that the next
generation of servers emerging in 2012 is expected to fe
ature dual port 10 GbE LANs
on the Motherboard (LOM) based on 10GBASE
-
T. 10GbE server connectivity will
also provide support for unified storage networking based on NAS, iSCSI, and FCoE.



I/O Virtualization

I/O virtualization supports a number of virtual Et
hernet connections over a single
physical port. This allows the assignment of a virtual GbE port to each VM, a
recommended best practice by VMware.

The combination of flatter network designs and a transition of server connectivity from GbE
to dual 10 GbE
will result in a much greater emphasis of the scalability and port density of 10
GbE and 40 GbE data center switches.


4



Scalability of Popular Two Tier Data Center LAN Designs


The scalability of a LAN architecture is determined by the number of server port
s that can be
supported with a given level of redundancy and over
-
subscription at different points within
the LAN topology. Many data center LANs being deployed today are based on a two tier
design that provides high levels of redundancy and low over
-
subsc
ription levels for server
-
to
-
server traffic. Two tier LAN designs are frequently implemented with Top of Rack (TOR)
access switches in conjunction with chassis based aggregation switches. The aggregation
switches are connected to the LAN core and to the In
ternet, but all the server
-
to
-
server traffic
within the data center flows only through the two tiers of access and aggregation switches.


Figure 1 shows a general model for two tier switched LANs that takes into account both
connections for redundancy and
connections to the LAN core. It is assumed that all servers
are attached to the access/TOR switches via 10 GbE ports. Any inter
-
switch links at the
access layer are assumed to be 10 GbE, and all other inter
-
switch links (i.e., inter
-
aggregation,
access
-
to
-
aggregation and aggregation
-
to
-
core) are assumed to be 40 GbE. If a given model of
switch does not yet support 40 GbE, a LAG with four 10 GbE member links could be
substituted. It should be noted, however, that a 40 GbE link is preferable to a LAG of four

10
GbE links because having a single 40 GbE link avoids the issues that can occur when
attempting to load balance traffic that consists of a small number of high volume flows.




Figure 1: Scalability Model for Two Tier Data Cen
ter LANs



5

This model can be applied equally well to two tier LANs based on multi
-
chassis LAGs and
two tier fat trees. The model focuses on P, the number of 40 GbE ports per aggregation switch
and the number of ports required to make connections both within

and among network tiers.


In the model,
Ccore

is the number of 40 GbE ports per aggregation switch that are used to
connect to the LAN core,
Cagg

is the number of 40 GbE ports per aggregation switch that are
used to connect to other aggregation switches
(e. g., for ISL/VSL). There may also be 10 GbE
inter
-
switch links within the access/TOR tier to support virtual switch/router functions such as
multi
-
chassis LAG (MLAG) or VRRP.


The access/TOR switches may be oversubscribed with more switch bandwidth all
ocated to
server connections vs. the amount of bandwidth that is provided from the access tier to the
aggregation tier. The over
-
subscription ratio is given by the following ratio:


The amount of bandwidth allocated to server access/The amount of bandwidth

allocated to access
-
to
-
aggregation connectivity)


Throughout this white paper, the over
-
subscription ratio will be referred to by the letter
m
.


A typical high density TOR switch has 48 10 GbE ports for server connectivity and four 40
GbE ports for inter
-
switch connectivity. Where servers are single
-
attached to these TOR
s
witches, m is equal to (48 x 10)/(4 x 40) = 3. Where the servers are dual
-
attached to a pair of
TOR switches with active
-
passive redundancy, m = 3, but the effective over
-
subscription ratio
is 1.5:1 because only one of the pair of server ports is active at

any given time. Where the
servers are dual
-
attached to a pair of TOR switches with active
-
active MLAG redundancy, the
requirement for inter
-
switch connections (
Cacc
) between the TOR switches means there are
two fewer 10 GbE ports per TOR switch available
for server connectivity and the over
-
subscription ratio is equal to m = (46 x 10)/(4 x 40) = 2.88



As shown in Figure 1, the number of 40 GbE ports per aggregation switch that is available for
connections to the access layer is equal to P
-
Ccore
-
Cagg and t
he number of 10 GbE access
layer ports that are available for server connection per aggregation is equal to 4 x m x (P
-
Core
-
Cagg). For a two tier LAN design with multiple aggregation switches, the number of
available server ports is 4 x S x m x (P
-
Core
-
Ca
gg), where S is the number of aggregation
switches.


It should be noted that the model presented in Figure 1 is based on having a single
aggregation switch, and the factor S needs to be included to account for an aggregation tier
with multiple aggregation
switches. For fat trees, the number of aggregation switches, or
spine switches, is limited by the equal cost forwarding capabilities (16 paths is a typical limit),
as well as the port density P. The port configuration of the access/TOR switch also imposes

some limitations on the number of aggregation/spine switches that can be configured. For
example, for a TOR switch with 48 10 GbE ports and four 40 GbE ports the number of 40
GbE aggregation switches is limited to four. Scaling beyond S=4, requires both a

denser
access switch with more 40 GbE ports and more 10 GbE port as well to maintain a desired

6

maximum over
-
subscription ratio. The ultimate fat tree scalability is attained where the 10
GbE/40 GbE access switch has same switching capacity as the aggregat
ion/spine switches.


With these caveats, the model takes into account redundancy and scalability for various Layer
2 and Layer 3 two
-
tier network designs as summarized in Table 1.




Parameter

2 Tier L2


2 Tier Layer 2

MLAG

2 Tier Layer 2

Fat Tree

2 Ti
er Layer 3

Fat Tree

Redundancy

none

full

full

Full

Ccore

variable

variable

variable

variable

Cagg

0

ISL/VSL

2 per agg
switch

0

0

Cacc

0

active/passive
server access: 0

active/active:

2 per TOR

active/passive
server access: 0

active/active:

2 per T
OR

active/passive:

2 per TOR
active/active:

2 per TOR

Max 10 GbE
server ports

4Sm(P
-
Ccore
-
Cagg)

S=1

4Sm(P
-
Ccore
-
Cagg)

S=2

4Sm(P
-
Ccore
-
Cagg)

S = # of
aggregation
switches

4Sm(P
-
Ccore
-
Cagg)

S = # of
aggregation
switches

Scaling

Larger P, m

Larger P, m

L
arger P,m,S

Larger P,m,S


Table 1: Scalability of Two Tier 10/40 GbE Data Center LANs


As highlighted in Table 1, the only way that the scalability of the data center LAN can be
increased is by increasing the:




Number of aggregation switches



Number of 40
GbE ports per aggregation switch



Level of over
-
subscription


As stated earlier, a typical initial design process might start from identifying the required
number of server ports, the required redundancy, and an upper limit on the over
-
subscription
ratio. A
s shown in Figure 2, calculating the required number of 40 GbE ports per aggregation
switch to meet these requirements is accomplished by inverting the scaling formula. The way
this formula can be used is for an IT organization to:


1.

Determine required num
ber of server ports

2.

Select the desired network type from Table 1. This will determine Cagg

3.

Select
an

access/TOR switch model. This together with the network type will determine
Cacc and m.


7

4.

Select the desired Ccore. This will determine over
-
subscription rat
io for client/server
traffic via the core

5.

Calculate the required port density of the aggregation switch



Figure 2: Required Aggregation Switch Port Density

for Two Tier 10/40 GbE Data Center LANs



To exemplify the formula show
n in Figure 2, consider the following network parameters:




The number of servers ports = 4512



M = 3



S = 2



Ccore = 2



Cagg = 2



The formula in Figure indicates that in order to support the indicated network parameters, an
access switch with 192 40 GbE ports

is required.


Figure 3 shows an example of a data center network that provides fully redundant Layer 2
server
-
to
-
server connectivity based on 94 TOR switches, each having 48 10 GbE ports and 4
40 GbE ports plus a pair of high density aggregation switches
with 192 40 GbE ports each.
The topology is an MLAG Layer 2 network with oversubscribed TOR switches. Each of the
2,256 servers is connected to two TOR switches in an active/passive mode. The same
configuration could also support 4,512 single
-
attached serv
ers. With active/passive
redundancy, the over
-
subscription of access switches for server
-
to
-
server traffic is 1.5:1.




Figure 3: Redundant Two Tier Network Configuration


8


For active
-
active server connectivity, each pair of TOR
switches would need to be configured
as a virtual switch with a pair of inter
-
TOR 10 GbE links for the ISL/VSL connectivity
required for the virtual switch, as shown in Figure 4. This would reduce the number of servers
per TOR switch from 24 to 23 and the
number of dual
-
attached servers to 2,072. With
active/active redundant MLAG server connectivity, the over
-
subscription ratio for server
-
to
-
server traffic is 2.88:1.







Figure 4: Redundant Two
-
Tier Network Configuration


Buildi
ng a comparable network with essentially the same number of 10 GbE server ports and
similar over
-
subscription ratios using similar TOR switches and an aggregation switch with
half the density (i.e., 96 40 GbE ports) requires some design changes. Comparing

the two
designs provides an illustration of the effect that the density of the aggregation switch can
have on the network design and the resulting TCO.


One possibility would be to build a Layer 2 fat tree network using four aggregation switches
in the sp
ine/aggregation layer and the same number of TOR switches (94) as the leaves/access
switches. However, most TOR switches do not yet support Layer 2 equal cost multi
-
path
forwarding alternatives other than with some form of MLAG. One workaround is to move
the
Layer 3 boundary from the aggregation switch to the TOR switch and build a Layer 3 fat tree
with OSPF ECMP providing the multi
-
path functionality. Figure 5 shows what this could look
like. Here the ISL links are at the TOR level rather than the aggrega
tion level and the server
connection can be made active/active without affecting the topology. With active/passive
redundancy, the over
-
subscription of aggregation switches for server
-
to
-
server traffic is
1.44:1, while with active/active redundant server c
onnectivity, the over
-
subscription ratio is
2.88:1. Note that a Layer 2 and Layer 3 fat trees based on switches with the same port
densities at the aggregation and access levels have the same physical topology.


9




Figure 5: Redun
dant Two
-
Tier, Layer 3 Fat Tree


If a TCO comparison is made of the two networks shown in Figures 4 and 5, some of the
differences to consider are the:




Capex and Opex differences with four switches vs. two at the aggregation level, including
switch cost,
power capacity requirements, rack space requirements, annual power, annual
routine admin, and annual service contract costs



Difference in the number of server ports per TOR



Differences in over
-
subscription ratios to the core



Eight links vs. four links to t
he LAN core needed for redundancy



Administrative cost and complexity differences with 98 Layer 3 devices vs. two Layer 3
devices


In addition, in a Layer 3 fat tree, there is a need for pre
-
standard VXLAN, NV
-
GRE or some
other virtual networking solution t
o enable VM migration across Layer 3 boundaries
.


This example shows some of the complexities that can be encountered in comparing the
TCOs of competing data center switching solutions that are based on switches of different
port densities, as well as somew
hat different functionality.

A Data Center LAN Cost Model

A typical design process for a data center LAN would include an analysis of the client/server
and the server
-
to
-
server traffic flows, resulting in a preferred topology as well as the required
number

of server access ports, permissible over
-
subscription, required redundancy, and
required connectivity of the data center to the campus LAN and/or Internet.


10

Adding redundancy at the server access level (e.g., by taking advantage of dual port 10 GbE
LOMs)
and connecting each server to two physical access switches doubles the number of
server access ports required. Similarly, adding redundancy at other layers of the hierarchy can
essentially double the number of ports required at those levels.

The formula in

the preceding section can be used to estimate the total number of 10/40 GbE
ports that are required by the design and the number of aggregation and access switches.
These estimates can then be used to develop Total Cost of Ownership (TCO) comparisons
betw
een different models of switch or different two tier switch topologies.

The equations in Figure 6 provide a simple example of a TCO analysis for a greenfield data
center network requiring all new switches, floor space, and power/cooling facilities. This
an
alysis focuses on the comparison of competing models of switch in a similar topology. For
simplicity, costs of cabling and some other costs are not considered.


Figure 6: A Simple TCO Model


According to the Uptime Institute
3
, th
e cost of provisioning a kilowatt of UPC power/cooling
for a Tier II data center is $12,500, while the corresponding figure for a Tier III data center is
$23,000. Their estimate for the corresponding cost per square foot of rack space is $300. The
cost a d
ata center rack is estimated to be $3,500. Cost per kWhr is estimated at $0.073, and



3

Uptime Institute White Paper: C
ost Model $/kW and $/sq ft of Computer Floor


11

the cost of annual routine maintenance per Layer 2 switch is estimated at $360. The cost of
annual routine maintenance per Layer 3 switch is estimated at $720.

Based on th
e TCO model presented in Figure 6 and the above cost parameters, a TCO
comparison was made between two data center LAN designs. One design is the network
shown in Figure 4 and implemented with two 8
-
slot Extreme Black Diamond X
-
8 switches
with 192 wire sp
eed 40 GbE ports and 94 Extreme Summit X670V TOR switches. The
second design is shown in Figure 5 and is based on a competing aggregation switch (Switch
X) with 96 wire speed 40 GbE ports and TOR switches of the same port density as the
X670V. The compar
ison is based on a greenfield data center network with a requirement of
4,144 total server ports, active
-
active dual 10 GbE server connectivity and an over
-
subscription ratio for server
-
to
-
server traffic of 3 or less.



Number of
switches

Cost/

switch

44U
Racks/
switch

Power/

port

Power/

Chassis

Annual

Service

BD X
-
8

2

$802,900?

14.5/44

W

2,52 kW


Summit
X670V

94

$25,000

1/44


257W


Switch X

Aggregation

4

$

25/44

W

2.82 kW


Switch X
TOR L3

94

$44,000

1/44


199W


Table 2: TCO Parameters for Comparison o
f Two Network Solutions for 4.144 Server
Ports



Switches

Racks

Tier II
Power
Capacity

Space

Annual

Power

Routine
Admin

Service

TCO

Extreme









Switch X









Difference









Table 3: Results of TCO Comparison of Two Tier Network with Aggrega
tion Switches
of Different Densities

In Table 3, the TCO was reflects OPEX NPV calculated based on a 5 year lifetime and an
interest rate of 4%.

Summary

An estimation of the capital cost of a new or upgraded data center LAN needs to take into
account more

factors than simply the cost per port of LAN switches being considered. A
preferred approach is to perform a TCO analysis based on a preliminary network design
taking into account the total number of server access ports required, required limits of over
-
s
ubscription for server
-
to
-
server traffic, the port density of the switches, and the total number
of switches and switch ports required.


12

As has been shown, the port density of the switch can have a significant effect on the LAN
topology, the level of over
-
s
ubscription, the total number of switch ports, and the network
power consumed in order to support a required number of server access ports. This
underscores the fact that IT organizations that are updating their data center LANs should
place greater emphas
is on the port density of LAN switches as a differentiating factor among
competing products.

Beyond switch cost, other significant CAPEX factors include the cost of power/cooling
capacity required to support the switches deployed and the physical density o
f the switches
which affects the rack space and square footage consumed by network hardware.

The port density of switch, together with the number of switches required, can also directly
affect OPEX in terms of routine administrative costs, power consumpti
on, and maintenance
contract costs.