IP NGN Operating systems and HW recap

thoughtlessskytopNetworking and Communications

Oct 29, 2013 (3 years and 9 months ago)

69 views

IP NGN Operating
systems

and HW
recap


IOS, IOS
-
XE, IOS
-
XR:


M
ain differences, pros, and cons


Targeted hardware platforms:


IOS: 7200, 7600/6500, ME


IOS
-
XE: ASR 1000, ASR900, Catalyst 4500E Sup7


IOS
-
XR: ASR 9000, CRS
-
1/3, XR12000


SP Hardware platform comparison:


Flexibility and Features


Performance: CPU forwarding
vs

Network Processor
vs

“Traditional” ASIC


Scalability: Centralized
vs

Distributed


Maintainability/Resilience: Monolithic, Modular


Describing Software Architecture and Features


Cisco IOS XE


Cisco IOS XR


Software
Maintenance
Operations:


O
n
Cisco IOS
XR


On Cisco
IOS XE


Explaining Configuration Management with Cisco IOS XR Software


Ethernet, OSPF, IS
-
IS, BGP on IOS/XE, XR


IOS, IOS
-
XE, IOS
-
XR: Main
differences
, pros
and cons: IOS.

Main “pros”


IOS is old (yes, that can be considered a good thing):


the most well
-
known Switch/Router OS amongst Networking professionals.


one of the most complete Router OS: extremely feature
-
rich (well that will depend on the hardware)


One OS for a large range of devices


Main “cons”


IOS is old (and therefore inherits an “old” design)


IOS is monolithic, completely adherent to the hardware, and does not provide any kind of isolation between
“processes”, neither from a CPU nor memory point of view:


V
irtual memory is shared by all IOS processes: nothing prevents buffer overflows.


S
cheduler is non
-
preemptive: if SNMP decides it should keep CPU busy, it can, and other processes (BGP…) will be prevented
from running.


You cannot upgrade IOS (or parts of it) without disruption unless you are

running
expensive

dual
-
supervisor

hardware


The «

One OS for a large range of
devices

»up
there

is

not far
from

a lie:
even

if
it

appears

somewhat

alike
, a 7200 IOS and a
3750 IOS have not
much

to
share
.


IOS, IOS
-
XE, IOS
-
XR: Main
differences
, pros
and cons: IOS
-
XE (1)

Mains “pros”


IOS
-
XE keeps IOS running on top of a Linux kernel, as a process.


Keeps the same feature
-
set
as the one
available

in the IOS train
that

was

embedded
.


An IOS
-
like look
-
and
-
feel on the command
-
line for most of the configuration / management.


IOS
-
XE allows for separation between Hardware (managed by
linux
) and IOS
itself:


Being abstracted from (some) hardware details allow for more convergence and coherency
between development teams and lessen the dependency and work needed for each
hardware
-
specific releases.


Allow for “software redundancy”, that is, running two IOS processes on the same hardware,
behaving as if there was two Sup cards, allowing ISSU (almost, see below).


Depending of the hardware infrastructure underneath, allow to upgrade some parts
(packages) of the Router OS, without disruption on other parts.


Having multiple packages allows to run them at different places: FE package runs on the FE
CPU, main package on the RP CPU, and interfaces drivers and management package runs on
the SIP. This allows to helping to scale and prevents SPOFs.

IOS, IOS
-
XE, IOS
-
XR: Main
differences
, pros
and cons: IOS
-
XE (2)

Main “cons”


IOS
-
XE keeps IOS running on top of a Linux kernel, as a process.


Hence there is no more separation between the processes running inside IOS
than in the traditional one.


The modularity is not
that

extensive:


IOS is one “module” or “package”


Other modules are hardware
-
specific parts (FE, interfaces, main
linux

kernel
underneath…)


On an ASR 1000, there is 7 packages. We are far from a real
snmpd
/
bgpd

isolation. And
no, you cannot ISSU the kernel on a single
-
sup.

IOS, IOS
-
XE, IOS
-
XR: Main
differences
, pros
and cons: IOS
-
XR (1).

Mains “pros”


IOS
-
XR is really a new and modular Unix
-
based Operating System, conceived for scale:


Based on a QNX micro
-
kernel: everything except core CPU/RAM management, IPC, and scheduling is
offloaded to modules: allowing to update drivers without touching at the running kernel


Allows a completely distributed platform: running a
microkernel

and
modular

distinct
processes

for
each

feature

forces
you

to use IPC and APIs:
looking

at

another

process

memory
is

not possible. Once the switch to
IPC
instead

of
shared

memory
is

done
,
distributing

processes

becomes

much

more
easier
.


IOS
-
XR allows for separation between Hardware (managed by QNX modules), Kernel, and IOS
processes :


Being abstracted from (some) hardware details allow for more convergence and coherency between
development teams and lessen the dependency and work needed for each hardware
-
specific releases.


Depending of the hardware infrastructure underneath, allow to upgrade some parts (packages) of the Router
OS, without disruption on other parts.


Exemple

of that is patching a critical security bug on, let’s say, BGP, without going through a full upgrade at
that time.


IOS
-
XR is not
that

new
:


development

started

in 1997 (IOS
-
NG) and
was

released

in 2004 for CRS
-
1, in 2005 for 12000, and in 2009 for
ASR9000,
so

there

is

some

history
.

IOS, IOS
-
XE, IOS
-
XR: Main
differences
, pros
and cons: IOS
-
XR (2).

Main “cons”


IOS
-
XR
have
some

differences

on the configuration point of view, and a lot
of differences on the software management.


Layer 2 configuration went from switch
-
like configuration to EVC model (yes, on an
SP network, that is a good thing, but you have to train teams)


Layer 3 route filtering and mapping now use the RPL syntax (same commentary)


The modularity is not
that

magic:


Yes, security patches are upgradeable one by one independently of supporting OS, but that’s
temporary, and of course is only supported when the APIs don’t change.


You still end up upgrading the whole router (so as to keep versions consistent) in the end
when new features are needed.


This modularity have a cost: the time of

a simple 3
-
steps upgrade of downloading one image,
updating the boot variable, and rebooting are way behind

IOS:
Targeted platforms


IOS: legacy SP platforms still available now


7200: once everywhere in SP edge, now nearly abandoned


full software router: Extremely flexible, but:


no performance


no hardware
policers

/ protections.


Centralized: no scalability.


7600/6500: once the star in SP core and aggregation


distributed ASIC
-
based platform: Some performance, but not as much feature and flexibility
as a 7200.


Security features like IPSEC are available through dedicated add
-
on cards with limited
bw


ME: Carrier Ethernet Edge


basically a catalyst switch buffed up with router features on specific (Service) ports.


Following the EVC/EFP pragma, as ASR9000, allow key features like MPLS to the edge.


ASIC based, not scalable.

IOS
-
XE:
Targeted platforms


IOS
-
XE: SP edge, SP Aggregation


ASR 1000 series


7200VXR/10K successor


Network
-
Processor based:


lots of Feature (even Software
-
typical ones like PPP or IPSEC
vpn
)


Lots of Performance (the bigger NP (ESP100) able to do 100Gbps routing including 30Gbps
ipsec
)


Easy to implement new features: NP is coded with ANSI
-
C, not ASIC
-
specific code.


Centralized: not much scalable: now 100Gbps max.


From 1ru/5Gbps/1spa/no redundancy to 13ru/100Gbps/6sip(24spa)/full
hw

redundancy


Ability to scale FIB
(
going over the 1M routes in FIB cap)


ASR 903


Catalyst ME successor


ASIC
-
based: more performance than feature


not
only

Ethernet
though

(E1/T1,
extended

clocking
)


some

mobile
backhaul

features

(RAN)

IOS
-
XR:
Targeted platforms (1): ASR9000


IOS
-
XR: SP core, SP Aggregation, SP edge


ASR 9000 series


12000

successor


Network
-
Processor based:


L
ots of Feature but geared toward Core features (less flexible than ASR1000 NP. For example LNS or IPSEC
vpn

are not available)


Some software
-
like features (CGN) are available through an
addon

card (ISM)


Lots of Performance, depending of the architecture of the line card and the fabric (see below)


Distributed Data
-
plane:


Forward
-
Engine on each
linecard


one or two RSP comprising RP + Fabric


bw

per slot between 90Gbps (1xRSP), 440Gbps (2xRSP440) on 9010 and 9006


b
w

per slot 550Gbps on 9922 (Fabric offloaded from RSP and put in dedicated FC cards)


From 2ru/120Gbps/2mpa/no redundancy to 43ru/48Tbps/20LC/full
hw

redundancy


Ability to scale FIB
(
going over the 1M routes in FIB cap)


Virtualization features:


MCLAG:
providing

another

device

the «

illusion

» of
being

one router
through

identical

LACP
announces

and
synchronization

between

the
two

ASR9000
routers
, in an active/standby
fashion


nV
:


Allows

clustering

two

ASR 9000
together

with

dedicated

EOBC links and a bundle of up to 32 10/100G data ports.


allows

connecting

a satellite (9000v, a
standalone

linecard

[44x1g, 4x10g
uplinks
]) in a
remote

location.

IOS
-
XR:
Targeted platforms (2): CRS1/3


IOS
-
XR: SP core, SP Aggregation


CRS1/3 series


From single
-
shelf, 4 slots, 320Gbps up to
multishelf
, 1152 slots, 322Tbps


About the same level of features than ASR 9000, with much more scale, but lower bandwidth per slot


Use of specific cards for software
-
like features (CGSE for CGN)


Fully distributed (
Dataplane

AND

Control
-
Plane):


Instead of a usual
FE+interfaces

linecard
, the two features are separated, 8 of each per chassis)


MSC (
MultiService

Cards) and FP (Forwarding Processor Card) are distributed FE cards, paired with an
interface module, they can handle 40(CRS1) or 140(CRS3)
Gbps

of traffic, depending on the MSC/FP model.


PLIM: Interface modules


FC: Fabric Cards are handling traffic between MSC / FP (8 FC/Switch Fabric Modules per chassis)


Route processors (2 per chassis, Active/Standby)


Distributed Route Processors sitting in MSC+PLIM slots allowing more than 2 RP, so you can:


Dedicate a pair of them to create an SDR (hardware virtual router, with dedicated LCs)


Use more than two RP to distribute routing processes


In a
multichassis

system:


Y
ou have chassis dedicated to Fabric switching, others containing MSC/FC + PLIMs


Some more cards to allow for EOBC communication between the chassis

IOS
-
XE/XR:
Targeted platforms (3): others


IOS
-
XR


XR 12000


From 2.5G per slot, 6 slots, 5RU to 40G per slot, 16 slots, 40RU


About the same level of features than ASR 9000, with lower bandwidth


Use of specific cards for software
-
like features (ISM for )


Fully distributed (
Dataplane

AND

Control
-
Plane):


Instead of a usual
FE+interfaces

linecard
, the two features are separated, 8 of each per chassis)


MSC (
MultiService

Cards) and FP (Forwarding Processor Card) are distributed FE cards, paired
with an interface module, they can handle 40(CRS1) or 140(CRS3)
Gbps

of traffic, depending on
the MSC/FP model.


PLIM: Interface modules


FC: Fabric Cards are handling traffic between MSC / FP (8 FC/Switch Fabric Modules per chassis)


Route processors (2 per chassis, Active/Standby)


Distributed Route Processors sitting in MSC+PLIM slots allowing more than 2 RP, so you can:


Dedicate a pair of them to create an SDR (hardware virtual router, with dedicated LCs)


Use more than two RP to distribute routing processes

IOS
-
XE/XR:
Targeted platforms (4): others


IOS
-
XE


Catalyst 4500E Sup7E


ASIC
-
based: more performance than feature (48Gbps per LC)


Centralized: not much scalable (848Gbps max on the FE).


From 7U/no redundancy/2LC to 14RU/redundancy/8LC


Number of routes in FIB limited to 60k to 256k

IP NGN Operating
systems

and HW
recap


IOS, IOS
-
XE, IOS
-
XR:


M
ain differences, pros, and cons


Targeted hardware platforms:


IOS: 7200, 7600/6500, ME


IOS
-
XE: ASR 1000, ASR900, Catalyst 4500E Sup7


IOS
-
XR: ASR 9000, CRS
-
1/3, XR12000


SP Hardware platform comparison:


Fl exi bi l ity and Features


Performance: CPU forwardi ng
vs

Network Processor
vs

“Tradi ti onal” ASIC


Scal ability: Central i zed
vs

Di stri buted


Mai ntai nability/Resilience: Monol i thi c, Modul ar


Describing Software Architecture and Features


Cisco IOS XE


Cisco IOS XR


Software
Maintenance
Operations:


O
n
Cisco IOS
XR


On Cisco
IOS XE


Explaining Configuration Management with Cisco IOS XR Software


Ethernet, OSPF, IS
-
IS, BGP on IOS/XE, XR


IOS XE Software architecture

IOS XE Software architecture:

Software
redundancy

Cisco IOS
-
XE Packages (ASR1000)

IOS XE Software architecture
related

to HW

IOS
-
XR Software Architecture


QNX
Microkernel

based
: the
only

non
-
restartable

parts are the
Timer
,
Scheduler
,
IPC, and Memory management.


Each

«

function

» in the router
runs

as a
process
,
with

its

own

virtual

memory and
scheduled

in a
preemptive

way
: no more CPU HOG or buffer
overflows
.


This
allows

process

restartability

up to
very

low

level
:


Drivers


TCP/IP
Stack


Forwarding
,
ACLs
, BGP, etc.


But
it’s

not
magic
:
there

can

be

dependancies

between

processes

requiring

to
restart more
processes

than

initially

planned
.


Having

separate

processes

requires

you

to have an efficient
way

of
exchanging

data,
may

it

be

between

processes

(
using

IPC) or for
delivering

packets

to
processes

(
using

LPTS [Local
Packet

Transport Protocol], an HW
policed

and
stateful

packet

delivery

mechanism
)

IOS
-
XR Software Architecture

Fully

distributed

on multiple RP
and
chassis
.

Hardware Virtual Router
capable on multiple
chassis
.

IOS
-
XR Software Architecture
related

to HW

IOS
-
XR Software architecture: packages (1)


IOS
-
XR Image:


Base Image (Mini):


OS


Admin


Forwarding





PIEs


Unique PIE for
each

feature
,
including
:


Multicast


MPLS


Manageability


Security


Each

critical

bugfix

gets

released

as a PIE
called

an SMU for emergency upgrade.


IOS
-
XR
runs

two

kinds

of code: PD (Platform
Dependant
) and PI
(Platform
Independant
).


IOS
-
XR
is

using

packages to manage software.
These

packages are
called

PIE (Placement
Independant

Executables
)

IOS
-
XR software architecture: packages (2)

IOS XE/XR Unique
features

on main line
products
: ASR 1000/9000

IOS
-
XE / ASR 1000


IPSEC
termination

/
VPNs


ISG: multiple
subscriber

protocols

termination

and large
-
scale

LNS


OTV:Datacenter

Interconnect


NPS:Network

Positionning


NAT/Firewall

IOS
-
XR / ASR 9000


Carrier Ethernet (OAM)


BNG:
only

does

LAC or
IPoE
/
PPPoE

termination


Video

Monitoring


Network
Virtualization
:


Clustering


Satellite


Real
modularity

IP NGN Operating
systems

and HW
recap


IOS, IOS
-
XE, IOS
-
XR:


M
ain differences, pros, and cons


Targeted hardware platforms:


IOS: 7200, 7600/6500, ME


IOS
-
XE: ASR 1000, ASR900, Catalyst 4500E Sup7


IOS
-
XR: ASR 9000, CRS
-
1/3, XR12000


SP Hardware platform comparison:


Fl exi bi l ity and Features


Performance: CPU forwardi ng
vs

Network Processor
vs

“Tradi ti onal” ASIC


Scal ability: Central i zed
vs

Di stri buted


Mai ntai nability/Resilience: Monol i thi c, Modul ar


Describing Software Architecture and Features


Cisco IOS XE


Cisco IOS XR


Software
Maintenance
Operations:


O
n
Cisco IOS
XR


On Cisco
IOS XE


Explaining Configuration Management with Cisco IOS XR Software


Ethernet, OSPF, IS
-
IS, BGP on IOS/XE, XR


Software maintenance
operation

on IOS
-
XE


Software
is

always

provided

as a single image,
you

can

keep

it

this

way


Or
expand

it

into

submodules

to
obtain

modularity

and issu


Can
be

upgraded

«

as
you

would

upgrade a 7200

», one
-
shot

chassis
-
wide
,
without

ISSU


(
still

basically

updating

a boot variable and
reloading
)


Hardware
redundancy

ISSU:


Dual
-
SUP and Dual
-
ESP: non
-
disruptive
except

for the
linecards
, one by one


Specific

«

issu

» command
-
set, simple,
less

than

5
steps

+
linecards


Software
redundancy

ISSU:


Single
-
SUP
can

be

partly

upgraded

with

ISSU (
only

the RPIOS/RPACCESS part)


Single
-
ESP upgrade
is

disruptive for the data
-
path


Non
-
identical
-
versions running on
Rpbase

and RPIOS are not
meant

to
be

the «

normal

» setup.
It’s

available

in
order

to
quickly

and non
-
disruptively

patch a single
-
sup system for a
security

/
critical

bug, but in the end,
doing

a full upgrade
is

still

necessary
.


Same

commands

that

used

for Hardware
redundancy
,
small

changes

Software maintenance
operation

on IOS
-
XR


Software
maintainance

on XR
can

be

managed
:



with

a all
-
in
-
one image


.
nv

image,
can

be

booted

by TFTP,
method

Called

TurboBoot


Have to
take

care of
dependancies


With

specific

parts images (
see

«

IOS
-
XR Packages

»)


Upgrade
procedure

is

NOT simple, and varies
greatly

depending

on the versions


Following

the Upgrade
procedure

in the Release notes
is

really

required
.


Partial upgrades:
SMUs


Timely

temporary

point fixes for urgent issues for a
given

package version


Fix

will

be

integrated

in the
subsequent

IOS
-
XR maintenance release


Implementation

change
only
, no interface (CLI,API,IPC) changes or new
features



Software maintenance
operation

on IOS
-
XR


PIEs

are a
delivery

mechanism

for
packages


Used

to
deliver


Major release


New
functionality

(3.8,
3.9, 4.0)


Maintenance release


SW fixes (3.8.1,
3.8.2)


SMU


Fix

for a
specific

bug



Includes

authentication

info



Installed

from

IOS XR
admin

mode


PIE install used once system is
operational


Packages can be added or upgraded


System performs sanity checks


3 phase install


Add


Copy package and unpack


Activate


Restart processes/nodes


Commit


Lock activated packages
through reload


De
-
Activate


Rollback


Remove



uninstall



.
vm

files are the
other

delivery

mechanism


.
vm

files are
bootable

images


Used

as the Initial Install for GSR
migration



IP NGN Operating
systems

and HW
recap


IOS, IOS
-
XE, IOS
-
XR:


M
ain differences, pros, and cons


Targeted hardware platforms:


IOS: 7200, 7600/6500, ME


IOS
-
XE: ASR 1000, ASR900, Catalyst 4500E Sup7


IOS
-
XR: ASR 9000, CRS
-
1/3, XR12000


SP Hardware platform comparison:


Fl exi bi l ity and Features


Performance: CPU forwardi ng
vs

Network Processor
vs

“Tradi ti onal” ASIC


Scal ability: Central i zed
vs

Di stri buted


Mai ntai nability/Resilience: Monol i thi c, Modul ar


Describing Software Architecture and Features


Cisco IOS XE


Cisco IOS XR


Software
Maintenance
Operations:


O
n
Cisco IOS
XR


On Cisco
IOS XE


Explaining Configuration Management with Cisco IOS XR Software


OSPF, IS
-
IS, BGP on IOS/XE, XR


Major configuration changes on IOS
-
XR (1)


New CLI Format and Configuration Modes


Config modes
include
:


Privileged

exec

mode (in an SDR)


Global config mode


Config
sub
-
mode


Admin

mode:
Chassis

operations
,
outside

of SDR


Admin

mode
is

newly

introduced

compared

to IOS


Admin

mode
allows

viewing

/
configuring

shared

resources


Fabric


Logical

Router


Package installation


Commit support:


archived

(id, label,
comments

and
commands
)


Rollback

support for one or more
commits


Config exclusive to block
simultaneous

commits


Commits

can

be

best
-
effort,

Major configuration changes on IOS
-
XR (2)


Pre
-
Config

ability


Pre
-
config

feature allows configuring physical interface before they are inserted into the router.


Preconfigured interfaces are not verified or applied until the actual interface with the matching location is
available.


Allows reduction down time and helps improve operational tasks.


Prior to the LC being inserted


Select the interface


Configure the timing (e.g. for SONET controller)


Configure the framing


Configure the IP address


Route Policy Languages replaces the route
-
maps/
pfx

list:


Scaling: Using route
-
maps could lead to 100k


1M lines of configuration (e.g. 1000s of BGP peers).


Modularity: Exploit modularity to reuse common portions of configuration.


Parameterization: For elements which are not exact copies of each other we can add parameterization ( think
variables ) to get further re
-
use.


Improved Clarity: No Silently skipped statements.


Conditional Statements: nested If , If
-
Then
-
Else


Boolean

Expressions:
nested

and
paranthesized

and, or, and not

Major configuration changes on IOS
-
XR (3)


EVC mode for L2 configuration


There is no “trunk” interface


XR doesn’t strip
vlan

tags by default


There is no more SVI (interface
vlan
), but BVIs inside a bridge domain


And that makes sense, since
vlans

are not global anymore !


An L2 service is defined by an EFP:


Ethernet Flow Point, a
subinterface

with “l2transport”


Flexible
vlan

matching:


Default, untagged, 802.1ad, 802.1q


1
st

level, 2
nd

level, exact or partial, ranges


VLAN rewriting:


Can translate, pop, push one or more tags


Always symmetric (we push on output when we popped on input)


You group L2 services into bridge domains and add “interfaces” (EFPs) into it.


To get something equivalent to an SVI,
create

a BVI and
include

it

as «

router interface

»
into

your

bridge
domain
.


Major configuration changes on IOS
-
XR (4)


Routing protocol main
config

changes


All routing protocols are VRF
-
aware (using “address
-
family”)


Static is now a protocol in itself, hence:


static routes goes into “router static”


Subdivised

by address
-
family, like other “router” sub
-
commands


OSPF interfaces are added:


into “router
ospf



area X” sub
-
commands


with “interface X” instead of “network
x.x.x.x

y.y.y.y

area X”


ISIS is now multi
-
instances, and have a new hierarchical CLI


BGP is now


Distributed


Using RPL (see above)


Not using peer
-
group anymore but:


Session Groups: Set of commands applicable to BGP session itself.


Address
-
Family Group: Contents of the neighbor session, Information exchanged in the BGP session


Neighbor Group: Bundling of different Session group and AF
-
group forms a
neighbour

group.