Real, Relevant, Surprising and Fresh: Cisco Brand (Static)

decisioncrunchNetworking and Communications

Nov 20, 2013 (3 years and 6 months ago)

66 views

© 2011 Cisco and/or its affiliates. All rights reserved.

1

High Performance Network
Analysis


Enterprise Operate Practice

Cisco Services

Andrew Wojtkowiak


Network Consulting
Engineer

High Performance Network
Analysis


© 2011 Cisco and/or its affiliates. All rights reserved.

2


Background

Cisco Services performed an assessment of the wired
infrastructure to serve as a holistic health check of the
University Corporation of Atmospheric Research
network


Goal of the assessment

To identify immediate remediation needs

Provide Opportunities for network improvement



© 2011 Cisco and/or its affiliates. All rights reserved.

3

High Level
Findings



Strengths and
Concerns

Background

And

Key Areas
Assessed

Executive
Level
Findings


Encompassing
Projects

Remediation

Steps



Looking
Forward

© 2011 Cisco and/or its affiliates. All rights reserved.

4


The High Performance Network Analysis (HPNA) was performed to assure the stability of
the core routing and switching infrastructure


Performed as a holistic network health check


Emphasis placed on Availability and Resiliency with the Campus environments


On
-
site interviews and data collection


Analyzed ~80 devices as part of the HPNA


Collected detailed network data such as topology diagrams, software, network
standards, protocols, etc…


© 2011 Cisco and/or its affiliates. All rights reserved.

5


Network Topology


Protocol Resiliency


Network Service Resiliency


Hardware and Software

© 2011 Cisco and/or its affiliates. All rights reserved.

6

© 2011 Cisco and/or its affiliates. All rights reserved.

7


Dedicated and professional network staff

Everyone we worked with was very open, professional and accommodating


Excellent Hardware and Software replacement strategies

Hardware and Software is kept up to date and staff is knowledgeable of bugs and vulnerabilities


Change Management Process

Well documented and followed change management process


Individualized tools for Network Management

Tools for deployments, configurations, backups, and management



© 2011 Cisco and/or its affiliates. All rights reserved.

8


Single Points of Failure

Increased risk of a pervasive network incident; scalability and availability concerns


Process Documentation

Lack of formal process to follow. No repeatable steps that all team members can use.


Global Configuration Templates

Templates will help reduce configuration inconsistencies and ensure services are configured according to
policy


Configuration Inconsistencies

Increased time to repair due to troubleshooting overhead; decreased network security; compliance risk


© 2011 Cisco and/or its affiliates. All rights reserved.

9


A few single points of
failure

TCOM switch for internet
connectivity

Foothills Lab secondary
switch

NWSC second switch


Major risk with TCOM

Higher latency backup


Foothills under
construction, second
switch in move


NWSC secondary switch
is being considered


Foothills and NWSC would
limit connectivity from
those locations to the rest
of the network.


© 2011 Cisco and/or its affiliates. All rights reserved.

10

© 2011 Cisco and/or its affiliates. All rights reserved.

11


Processes are well defined
by the individuals who
perform the tasks

Software and Hardware
replacement

Standards for implementing
new devices


No actual defined
documentation


Only certain people are
well versed in processes


Not easily reproducible


No defined steps for
changes


Allocate time to turn
processes into
documentation


Allocate someone to
review the documents


Keep them up to date as
they change.

© 2011 Cisco and/or its affiliates. All rights reserved.

12

1

2

3

1

5

2

15

Cisco 6500 Series Switches IOS

12.2(33)SXH3
12.2(33)SXH4
12.2(33)SXH8
12.2(33)SXI4a
12.2(33)SXI5
12.2(33)SXI6
12.2(33)SXI8a
All CatOS has reached End of SW Maintenance, and will
no
longer receive attention with regards to defect or security
vulnerability patching

8.6

67%

8.4

33%

Total CatOS Summary

8.6
8.4
© 2011 Cisco and/or its affiliates. All rights reserved.

13


Configuration standards
are adhoc; without formal
documentation


No way to perform
configuration compliance
to a template*


N
umber of configuration
inconsistencies and errors
(Protocol, Service, Security)



Network unpredictability


Potential increased
troubleshooting overhead
and operational difficulty


Prolonged loss of
connectivity and service
interruption to critical
applications


Increased exposure to
security vulnerabilities


Increased cost associated
with operating the network


© 2011 Cisco and/or its affiliates. All rights reserved.

14


HSRP inconsistencies


Partially configured
advanced spanning tree
features


Optimize/Standardize
Spanning
-
tree priorities


OSPF passive interface




Some routers do not have
a peer


Possible loops or rouge
switches influencing the
network


Routing updates are not
limited


Implement changes to the
network to remediate the
smaller configuration
inconsistencies


The standard templates will
assist in ensuring fewer
deviations from standard.


© 2010 Cisco and/or its affiliates. All rights reserved.

15

© 2011 Cisco and/or its affiliates. All rights reserved.

16


Three buildings connected
in a partial mesh topology


Collapsed connections to
each other


Port density growth at
N*(N
-
1) rate for every new
building


Lack of modularity and
scalability


Large fault domains
across all buildings


Network disruption and
outages


Increased troubleshooting
overhead



Quantifiable cost increase
in both capital and
operational expenditure



Additional Capital Expenditure
associated with running fiber

Nx
(N
-
1) = 12 Ports (
6

Links)

Cost to Add 4
th

Building

Additional Operational Expenditure
associated with design complexity

© 2011 Cisco and/or its affiliates. All rights reserved.

17

Current Topology
-

No Core


Fully
-
meshed distribution layers


Physical cabling requirement


Routing complexity

© 2011 Cisco and/or its affiliates. All rights reserved.

18

This leading practice hierarchical design
has been proven to:




Promote easy growth and ease of
troubleshooting




Reduce capital and operational
expenditure




Create small fault domains




Promote deterministic traffic flows




Enable logical and physical
topology mapping


Center Green

Dedicated WAN /
Internet Switch Block

Mesa Lab

Foothills

New Location

TCOM/FRGP

Research Networks

Firewalls

Internet

Dedicated Core

© 2011 Cisco and/or its affiliates. All rights reserved.

19


Monitoring facing the
Internet

Intrusion Prevention

SPAN Sessions to security
team


Extensive ACLs on core
switches


No Control Plane Policing
to protect devices



Limited methods to log
and account for network
incidents


Increased CPU usage on
switches


Create method to evaluate
internal ACLs routinely


Consider Control Plane
Policing for basic
router/switch services

Routing

Switching

© 2011 Cisco and/or its affiliates. All rights reserved.

20

Correlating business impact (risk
reduction) to ease of execution and
exemplar implementation time


Project List:

1) Remediate single points of failure

2)
Create, utilize and maintain global configuration
standard templates

3)
Create, utilize and maintain process
documentation

4)
Remediate configuration inconsistencies within
the network


More complex to implement

Low priority

0
-
6
months

9months

> year

Easy to implement

High priority

1

Must Do


Reduce Risk

Very Hard

Quick Wins


High Business Impact

Easy But Low Return

2

3

4

© 2011 Cisco and/or its affiliates. All rights reserved.

21

Thank you.