Juniper Confidential - Inside Analysis

wartrashyNetworking and Communications

Oct 26, 2013 (3 years and 7 months ago)

147 views

INTRODUCING QFABRIC

REINVENTING THE DATA CENTER NETWORK

Simon Gordon



Juniper Networks

Senior Product Line Manager
-

FSG/DCBU

sgordon@juniper.net

+1
-
408
-
242
-
2524

2

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

QFABRIC

IS REAL

QF/Interconnect

QF/Director

QF/Node (QFX3500)

3

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

TRENDS IN DATA CENTER



Multi
-
core (8
-
>16
-
>32,….128,…)



Virtualization and VMs

Server Trends



SOA, Web2.0



MapReduce, Hadoop, Grids

Application Trends



Convergence to 10GE



Enhancements to Ethernet

Interconnect Trends

Consolidation



Mega DCs; 400K sq ft



4K racks, 200K servers

DC Scale

Low O/S

East
-
West traffic

10/40/100 GE

QFabric

Any Service

Any Port

4

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

TODAY’S
ARCHITECTURE IS NON
-
TRANSPARENT

Ethernet

Servers

NAS

Multi Tier Network

Scale

Latency

Scale vs. Latency

Scale

Bandwidth

Scale vs. Bandwidth

Location dependent adjacencies between nodes

1

2

3

4

1

2

1

2

1

3

1

3

1

4

1

4

5

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net


SRX and vGW


Inter
-
DC connectivity


MPLS and VPLS


Virtual Control


Single, scalable fabric

QFABRIC


1 TIER

SRX5800

MX
Series

Remote
Data Center

One large, seamless resource pool

Servers

NAS

FC Storage

6

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

QFABRIC BENEFIT

Scale

Latency

Scale vs. Latency

Scale

Bandwidth

Scale vs. Bandwidth

Traditional


design

QFabric

Traditional


design

QFabric

3 Design Principles

Data Plane

Rich edge, Simple core

Everything is one hop away

Director Plane

Federated Intelligence

Only way to scale with resilience

Management
Plane

N=1

Operational model of

a single switch

A Revolutionary New
Architecture

8

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

DATA PLANE IN A SINGLE SWITCH

1. All ports are directly connected

to every other port

2. A single “full lookup” processes
packets

Data Plane

9

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

Director PLANE IN A SINGLE SWITCH


Single consciousness


All the ports are managed from
a single point

Director Plane


Centralized shared table(s)
have information about all ports

Management Plane

10

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

SINGLE SWITCH DOES NOT SCALE

…but eventually it runs out
of real estate.

After this, the network
cannot be flat.

Ports can be added to a
single switch fabric.

Sacrifice simplicity or…

change the scaling model

Choice:

11

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

SCALING THE SWITCH

Disaggregate

So, we separate the

line cards and supervisor cards
from the fabric.

And replace the copper

traces with fiber links.

For redundancy add

multiple devices.

QF/Interconnect

QF/Node

QF/Director

12

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

SCALING THE SWITCH

Disaggregate

So, we separate the

line cards from the fabric.

And replace the copper

traces with fiber links.

For redundancy add

multiple devices.

Enable large scale.

QF/Interconnect

QF/Node

QF/Director

13

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

SCALING THE DATA
PLANE

Data Plane

QFabric is faster than any Ethernet chassis switch ever built

1.
All ports are
directly connected

to every other port

2.
A single “full
lookup” at the
ingress QF/Node
device





3.
Blazingly fast:
Always under 5us
3.71us
(short cables)

QF/Node

QF/Interconnect

14

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

Backup

The single

active instance
limits scalability

Director Plane

SCALING THE
DIRECTOR
PLANE

Old Model

Active/Backup

Active

15

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

The intelligence
and state is
federated,

distributed
across the fabric

Director Plane

New Host

Address

SCALING THE
DIRECTOR
PLANE

Director and
management
services use a
scale out model

New Model

Services Oriented

QF/Director

16

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

SCALING THE MANAGEMENT PLANE

Management

Plane


Single point of
management


Extensive use
of automation


Familiar
operational
model

Managed as a single switch
-

N=1

QF/Director

17

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

Servers

QFABRIC

CONVERGENCE


THE END VIEW

Convergence

Fully Blended Fabric


Fibre Channel Services


Flexible ports FC/FCoE/E


Fully converged unified Network


Storage

FC

Servers

FCoE

FC

FCoE

18

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

Servers

QFABRIC

CONVERGENCE


2011

Convergence

FCoE Transit Switch


Converged Enhanced Ethernet


Standards based (CEE or DCB)


Provides Perimeter protection
with FIP Snooping.

FCoE
-
FC Gateway


Ethernet or
Fibre

channel
gateway with FC ports at

the QF/Node


Interoperates with

existing SANs


Storage

FCoE

FC

FCoE

SAN

FCoE

Servers

FCoE

FC

19

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net


Hardware

20

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

QFABRIC HARDWARE

QF/Interconnect

Connects all the QF/Node devices

QF/Node

Media independent I/O
ToR

device.

Can be run in independent or fabric mode

QF/Director

2 RU high fixed configuration

X86 based system architecture

21

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

QFABRIC HARDWARE


INTERCONNECT

Front View

Rear View


21 RU high 8 slot chassis


128 QSFP 40G ports


wire
speed


8 fabric cards
(10.24Tbps/chassis)


Dual redundant Director board


Redundant AC power supply


Front to back air flow


QF/Interconnect

22

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

FABRIC HARDWARE


QF/NODE

Front View

Rear View


1 RU high fixed configuration


48 SFP+/36 SFP ports


12 FC capable (2/4/8G) ports


4 * 40G fabric uplink ports (can
also operate in 10G mode)


Redundant AC power supply


Front to back air flow


QF/Node

48
SFP+/36 SFP
ports

4 QSFP+ ports

12 FC Capable ports

Will also operate as a

Stand Alone Switch

QFX3500

23

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

QFABRIC HARDWARE


DIRECTOR


2RU device


Has GE ports to connect to
QF/Node and interconnect
devices


Based on x86 architecture

QF/Director

24

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net


System Design

25

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

QFABRIC CONFIGURATION FOR SMALL DEPLOYMENT

40 GB

LEGEND

1 GB

Solution for 768
10GE/1GE ports

2 Fabric cards per
chassis (25% fill rate)

Redundant uplinks

QF/Interconnect

QF/Node
#1

QF/Node
#3

QF/Node
#2

QF/Node #16

QF/Director

26

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

40 GB

LEGEND

1 GB

QF/Interconnect

Solution for 6,000
10GE/1GE ports

1GE connections to
the Director cluster

40 Gig uplink from each
Node to Interconnect

QF/Node
#1

QF/Node
#
128

QF/Node
#3

QF/Node
#2

QF/Director

QFABRIC CONFIGURATION FOR LARGE DEPLOYMENT

27

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net



QFabric

Software

28

Copyright
© 2010 Juniper Networks, Inc. www.juniper.net


SYSTEM ARCHITECTURE EVOLUTION

Forwarding Engine

Route Engine

Management

Pizza Box

Forwarding Engine

Route Engine

Management

Chassis

Forwarding Engine

Forwarding Engine

Forwarding Engine

Management

QFabric

Forwarding Engine

Route Engine

Peer

Peer

Peer

Peer

Forwarding Engine

Route Engine

Matrix, Virtual Chassis

Forwarding Engine

Route Engine

Slave

Slave

Route Engine

Management

Master

29

Copyright
© 2010 Juniper Networks, Inc. www.juniper.net


QFabric

SOFTWARE STACK

Platform

Data Plane

Control Plane

Fabric Control

Fabric Administrator

Centralized

Distributed

Inventory

Topology

Troubleshooting

Fault

Management

APIs

Connectivity

Views

L2/L3 switch stack

Platform

Data Plane

Control Plane

Platform

Data Plane

Control Plane

30

Copyright
© 2010 Juniper Networks, Inc. www.juniper.net



Management

31

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


FABRIC DIRECTOR

SINGLE POINT for signaling
and configuration: CLI, SNMP,
NETCONF/DMI (XML ), SMI
-
S

Director

Director

Fabric RE

Director RE

Partition RE

Hypervisor

Fabric

Administrator

Control And
Management
interfaces for
QFabric

Simplicity

Single Box
management
paradigm

Standard

SNMP,
Syslog
,
NETCONF, CIM,
SMI
-
S


Virtualized

Hide CP and DP
components,
views, scale


Automation

JUNOS built in
Automation
capabilities

32

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


QFABRIC

MANAGEMENT STACK

Director

Inventory

Topology

Troubleshooting

Fault

Management

APIs

Connectivity

Views

Business Services

Data Center Orchestration

Junos Space

EMS/NMS/Apps

Signaling and Configuration:

CLI, SNMP, NETCONF/DMI (XML),
SMI
-
S

Platform

Data Plane

Control Plane

JUNOS SDK

Space
SDK

WSDL/SOAP, REST APIs

Platform

Data Plane

Control Plane

JUNOS SDK

Platform

Data Plane

Control Plane

JUNOS SDK

BSS:
Juniper +
Partners

33

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


Director

FABRIC SYSTEM CONTROL SCALE OUT

Node 0

Node 1

Node N


N
-
way compute cluster


Automatic balancing of compute load and connection traffic (both south and
northbound)


No redundant nodes / hot spares


all resources available for computation


Graceful degradation upon failure


Scale out: adding nodes in service, nodes automatically discover each other


SFC application logic dynamically upgradable while running

SFC

Director

Director

QFabric

RE

Director RE

Partition RE

Hypervisor

Fabric

Administrator

34

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


Performance and
simplicity

of a single switch

A REVOLUTIONARY NEW ARCHITECTURE

Scalability and resiliency

of a network

35

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net



Migration to
QFabric


Introducing QFX3500



36

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


MIGRATING TO QFABRIC

QFX3500

MX Series

EX8216

EX4200

SRX5800

QFabric

Pod 1

Pod 2

4

37

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


INTRODUCING QFX3500

Front view

Rear view

Wirespeed switching

1.28 Tbps , 960 MPPS switching

FCoE & Fibre Channel Support

FCoE Transit Switch & FCoE
-
FC Gateway

Ultra low latency

Sub microsecond

Low power

5Watts/port

Layer 2

FCoE
-
FC

Layer 3

38

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


PORTS

Rear view

4 QSFP+ ports





Roadmap (not available at FRS)

63 port 10GbE

48 port 10GbE

4 port 40GbE

6 port FC

42 port 10GbE

4 port 40GbE

12 port FC

36 port 10GbE

4 port 40GbE

12 port FC

48 port 10GbE

1 port 40GbE

48 SFP+/SFP ports











12 port FC; 36 port GbE

39

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


Direct
Attached/
Twinax SFP+ copper

1, 3, 5, 7 meter

SFP+ optical transceiver

USR, SR, LR

SFP
transceiver

SR, LR, 1000BaseT

SFP optical
transceiver

2/4G or 8G FC
-
SW

TRANSCEIVER SUPPORT

Rear view

FC SFP

10GbE SFP+

GbE SFP Optical

GbE SFP Copper

40

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


Direct
Attached/
Twinax SFP+ copper

1, 3, 5, 7 meter

SFP+ optical transceiver

USR, SR, LR

63 PORT 10GBE IN 1RU (2Q2011)

Rear view

48 x 10GbE SFP+

QSFP+

4 x SFP+

To Servers

4 x 10GbE

4 x 10GbE

3 x 10GbE

4 x 10GbE



Roadmap (not available at FRS)




12 FC Capable ports

41

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


PERFORMANCE & SCALE

Feature

Scale

Throughput

1.28 Tbps

Forwarding

960 MPPS

Latency

900 nanoseconds

Packet buffer

9MB shared

MAC address

960K

IPv4 routes

20K

Multicast groups

4K

Firewall

filter

1,500

Maximum power

320 Watts

Nominal power

200 Watts

Nominal power per port

~4 Watts

Depth

28”

Air flow

Front

to back

42

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net


SECURITIES TECHNOLOGY ANALYSIS CENTER (STAC)
TEST
RESULTS

Simulates Trading Transactional Performance

Description

Supply to Receive

Latency,
1 Producer to 5

Juniper
QFX3500/

IBM LLM

Cisco

4900M/29West

Cisco

Nexus

5010/29West

Voltaire

IB/IBM LLM

Highest Supply Rate

(
msg
/sec)

1,500,000

1,300,000

1,300,000

1,000,000

Mean (micro seconds)

9

15

14

8

Max (micro seconds)

23

30

33

47

Standard

Deviation

(Jitter)

0

1

1

1

The Juniper QFX3500 in combination with IBM server and middleware with
SolarFlare

NICs delivered the best performance to date for product combinations with 10GE switches.

This product combination delivered more messages faster with lower jitter than any other
audited report in the STAC library.

42

STAC
-
M2 Benchmarks™ v1.0 Highlights

43

Copyright
© 2009 Juniper Networks, Inc. www.juniper.net CONFIDENTIAL


QFX3500


UNIVERSAL TOR

Certify Once; Deploy Everywhere

QFX3500

Feature Rich

Full L3; VirtualControl; FC
Gateway; HA; VPN

Fabric Attach

Unique Value
Add to Scale

Converged I/O

DCB; FCoE
-
FC Gateway; FCoE
Transit Switch

FC/FCoE

Ethernet

FC SAN

Ethernet/
IP

TOR

Ultra Low Latency

<1µSec; Cut
-
through; 40G

44

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

CONVERGED I/O

FEATURE RICH

FABRIC ATTACH

CONVERGED I/O

ULTRA LOW LATENCY


QFX3500 Solution


10GbE/
FCoE

standard


FCoE

transit switch


FCoE
-
FC Gateway


Feature rich DCB (PFC, ETS, DCBX)


FIP Snooping

Converged I/O with CNA &
FCoE

45

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

FCOE TRANSIT SWITCH USE CASE (QFX3500)


Requirements


10GbE server access including Blade
servers with pass through or with
embedded DCB switch


Copper and/or fiber cabling


High availability


Dual homed to aggregation layer


>40 port per ToR switch


DCB support with FIP Snooping


QFX3500 solution


48 (63) ports wirespeed 10GbE


Copper DAC and SFP+ fiber support


Hardware & software HA


DCB & FCoE transit switch support


FCoE is standard on all ports


PFC, ETS, DCBX support


FIP snooping support


Interoperability with Qlogic, Emulex CNA

MX series


MCLAG or VC

or

EX8200 VC

LAG

LAG

FCoE
enabled

SAN

FCoE Transit Switch

Rack servers or Blade servers with CNA

LAG

46

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

FCOE
-
FC GATEWAY USE CASE (QFX3500)


Requirements


10GbE server access including Blade
servers with pass through or with
embedded DCB switch


Copper and/or fiber cabling


High availability


Dual homed to aggregation layer


>40 port per ToR switch


DCB & FCoE
-
FC Gateway support


QFX3500 solution


48 (63) ports wirespeed 10GbE


Copper DAC and SFP+ fiber support


Hardware & software HA


DCB & FCoE
-
FC Gateway support


FCoE is standard on all ports


PFC, ETS, DCBX support


12 port FC (2/4/8G FC) with FC license


Interoperability with Qlogic CNA, Emulex
CNA, Cisco & Brocade FC switch



MX series


MCLAG or VC

or

EX8200 VC

LAG

LAG

FC
SAN

FCoE
-
FC Gateway

Rack servers or Blade servers with CNA

47

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

FCOE TRANSIT & GATEWAY SWITCH USE CASE


Requirements


10GbE server access including Blade
servers with pass through or with
embedded DCB switch


Separation of management between the
LAN & SAN Teams


Gateway administered by SAN Team


ToR administered by LAN Team


Support for Blade servers with pass
through or with embedded DCB switch


QFX3500
solution


FCoE Transit Switch at ToR


FCoE
-
FC Gateway at EoR


EX4500 as Transit Switch


3rd party Transit Switches


In particular blade shelf embedded switches

MX series


MCLAG or VC

or

EX8200 VC

LAG

LAG

Rack servers or Blade servers with CNA

FCoE
-
FC
GW

FC
SAN

LAG

FCoE
-
FC Gateway

FCoE Transit Switch

48

Copyright
© 2011 Juniper Networks, Inc. www.juniper.net

FCOE TRANSIT & GATEWAY SWITCH USE CASE


Requirements


10GbE server access including Blade
servers with pass through or with
embedded DCB switch


Separation of management between the
LAN & SAN Teams


Gateway administered by SAN Team


ToR administered by LAN Team


Support for Blade servers with pass
through or with embedded DCB switch


QFabric

& QFX3500 Solution


FCoE Transit Switch at ToR


FCoE
-
FC Gateway at EoR


EX4500 as Transit Switch


3rd party Transit Switches


In particular blade shelf embedded switches

MX series


MCLAG
or VC

Rack servers or Blade servers with CNA

FCoE
-
FC
GW

FC
SAN

FCoE
-
FC Gateway