A Case Study

smashlizardsNetworking and Communications

Oct 29, 2013 (4 years and 2 months ago)

305 views

Use of BGP and MPLS VPNs:
A Case Study

Fred P. Baker

CCIE#3555

Contents


Current Network


The MPLS VPN project


Routing Objectives


What we did


How we tested

Current Network

Current Environment


Hub and spoke to 4 data centers


Sites do not in general connect to 2 data centers due
to cost and OSPF issues


Generally place servers by geography


You servers are in the data center your links are in


Mostly Frame Relay to ATM interworking with
some private lines


70 of some 350 remote sites have 2 links


ATM PVC dual mesh between the data centers


12000 agent location network done by MCI with
combination of DSL and Fractional T1


Address Space


10.0.0.0/8


Mostly inside


Some BP


192.168.0.0/16


Used all over


172.16.0.0/12


Extranet


167.127.0.0/16


Public address space


Used mostly by extranet


Some legacy inside

Core


ATM PVCs


2 10meg between each pair of data
centers


2 routers on the core


So 2 meshes

Allstate Core


10.0.0.0 address allocation

/11 for core 1 per data center


Allstate Data Center


Routing Protocol


Single OSPF AS


Cisco and OS/390 based routers only


Firewalls now static routed


Peer authentication soon

Remote sites


AT&T frame relay at the site


ATM into the data center


Some ISDN backup


A remote site is connected to a single data
center (for now)


Servers and applications tend to have
geographic affinity


Remote Site


Remote Site Switch Layer



Agent Broadband


10,000 locations


Connected via IPSEC VPN


WorldCom managed routers


NO split tunneling


IPSec Transport with GRE tunnel to Dallas
and Hudson


Agent PCs are 10.*.*.*


Agent access is via Allstate Internet Proxy

Overview

Agent Broadband in Data Center


Agent office

Internet/Extranet


We do not use the default route


There are 3 data center with ISP
connections


We code static routes to the firewalls (we
don’t trust firewalls running dynamic
routing protocols) and redist to OSPF

The project

The project


We use a single data network provider


This is a single point of failure of that
providers ATM/Frame networks


Add a second data provider


Initially to use for the dual attached sites


Then convert 1 of the core ATM meshes to
the second provider

Layer 2 vs Layer 3 provider


Frame Relay is layer 2 connectivity


The routers have a direct peering relationship


Many providers are offering Layer 3


Costs are the same or even less


MPLS VPN is the data transport


Many providers are using MPLS to move even layer 2
networks


You have a routing relationships with the provider not
with yourself


So More complex to configure and fix


Not a simple OSPF network anymore


Which one we picked


Layer 3…


DR becomes free do not need to run more PVCs to a
DR data center


The data center placement of servers assumption is
changing


Apps are being put to 1 DC


Also there is more site to site traffic than we expect


So we can reduce traffic on the ATM core


And increase response time


Do dual homed sites first convert 1 link to L3


Single homed late

MPLS VPN


VPN

A/
Site

1

VPN

A/
Site

2

VPN

A/
Site

3

VPN

B/
Site

2

VPN

B/
Site

1

VPN
B/
Site

3

CE
A1

CE
B3

CE
A3

CE
B2

CE
A2

CE
1
B1

CE
2
B1

PE
1

PE
2

PE
3

P
1

P
2

P
3

10.1/16

10.2/16

10.3/16

10.1/16

10.2/16

10.4/16

Route types


CE customer Edge


your router


run BGP to provider


Knows nothing about other customers or provider
routes


PE provider Edge


Knows about all local customer VPNS


Has multiple routing tables


P providers


Transport only


No customer routes

Routing objectives


Support load share from the home DC


Remote site goes direct to non home DC
over L3


Remote site directly to remote site


Reduce transit of the core


Support a L3 provider in the core replacing
1 ATM mesh


Do not use remote sites to transit traffic


Technical Objectives


Limit the number of bgp attributes used


Keep the remote site configuration simple


Do not inject the default route unless you
must


How to inject the Internet routes




Routing protocol design

Don’t forget the 3 rules of routing


Longest subnet mask


Lowest distance


Best metric

BGP features we used


As path


Path length filters


No export


Backdoor


If AS Paths are equal then router uses
eBGP route


How to route


Must look at the routes going BOTH ways


Routes to


Routes from


The routes you advertise drags traffic to you


The routes you take in is how you route back


We load share by having each router use a
different path, then send equal cost into IGP

Result


Use MPLS
VPN based
L3 provider


Remote sites
2
nd

link to L3


Each data
center
connects to
L3


Will not use
L3 to route
between DCs
due to QoS
concerns

Routing


Use BGP at remote sites


Can use OSPF with SOME providers but not all


BGP works much better


Each site is 1 AS


EACH data center is 1 AS


This allows us to put an L3 provider in later


BGP routes BETWEEN ASes


Address ASes from private space


This is ok because provider is a VPN

Route injection to/from BGP


Allstate Data Center


Explicit network statements to BGP


Redist BGP to OSPF


Remote site routes


Redist from OSPF


Decided that using network statements to complex


BGP routers send just default route to any switches


We will accept the extra LAN transit


Internet routes


Redist static

Internet routes


There will be non BGP L3 switches
between Inet and allstate core


Redist static into OSPF already


So just redist into BGP also


Put internet router in same AS as
datacenter (have to as no direct path)


Use sync


Send to L3 provider and to sites over L3


BGP to L3 provider (and then
remote sites


Data center side


Send data center /11s


Send internet routes


Take routes from L3 provider


Do not forward other eBGP learned routes


Remote site side


Send all local routes


do not forward other learned eBGP routes


Remember the no export to kill transit


Receive all routes


Want to take L3 when I can


DC to Remote site FR


Send all bgp derived routes


Do as prepend of the data center AS


This makes AS path =2 for DC on FR and
L3 paths


This makes AS Path=3 for DC to DC via
ATM core so site to remote DC traffic over
L3

Remote site to DC on FR


Do as prepend of 1 AS at remote end


Need this so FR and L3 paths have AS
Path=2 so we load share


Filter routes with AS Path >1


I only want to send the local site routes up the
FR link


Do not want DC to send transit traffic to site

IBGP in the remote site


Set next hop self


Routers must have a shared Enet


No redist of BGP to OSPF


So cant use sync so cant transit a L3 switch


Do not forward routes I learn via FR


Do not want a transit from L3 up the FR link


Do not want a transit to L3 from FR link


Set no export attribute on routes from DC over the FR
link


This prevents site from passing them to L3


Cannot AS path filter on IBGP because I want to pass
the DC route via iBGP


Why I use no export

Results

DC to DC


Each site
learns over
ATM
network with
AS Path = 1


Cannot route
over L3
provider

Remote site to non home dc


Non home DC
sent via L3 AS
Path = 2


Home data
sends via FR
AS Path = 3
due to prepend


Use if L3
down

non home dc to remote site


Non Home DC
learns remote
site routes from
L3


Home data
center sends
only the /11
summary


so longest match
says L3

home dc to remote site


Load share


Routes from L3
have AS Path =
2


Routes from FR
have AS Path =
2 due to prepend


So each router
uses eBGP route

remote site to home dc


Don’t care as
much about load
share


Routes from L3
have AS Path =
2


Routes from FR
have AS Path =
2 due to prepend


So each router
uses eBGP route

remote site to remote site


Use L3
network


Learn site
specific routes
directly from
site


Learn /11
summaries
from DCs

Agent routes


Only dual DC connected things that don’t use BGP


Many routes summarized as /19s


I get these from MCI as OSPF externals


Have not decided how to inject them


They go to two data centers for redundancy


So I need to send them via BGP


So a router will get an OSPF external from the local MCI
connection and the other data center via BGP


eBGP < OSPF so BOOM


Use backdoor on core routers to set distance on the
agent routes to > than OSPF


So if local MCI connection up use it, else transit core

Testing

Local Testing


Use 7 routers


1 remote site
OSPF route not
shown


Paths


iBGP at remote


L3


FR to home DC


Inter DC

CPOC


Cisco Proof Of Concept


In Raleigh and San Jose


Lab use is free (if you are big enough)


Send in specific test plan


Your SE goes in a week ahead of time


Lab is all setup when you arrive

Testing


Test migrations


Test routing


based on our policies


failovers


Measure convergence


Test a migration of a core ATM mesh to L3


Get some data and experience on the MPLS
side


Try multicast over MPLS/VPN

CPOC Network Diagram

CPOC Learnings


Inject all links both ATM core and L3 into BGP as they
will source pings


Turn sync off due to code defect


You must explicitly code send community in iBGP


If you reference a non
-
existent as
-
path statement NO
ROUTES


OSPF LSAs stay in the data base up to 90 minutes due
to timer jitter


This is a migration issue


Do lots of clear routes/clear ip bgp in the migration


Need to change the BGP timers as default convergence
is 3 minutes


iBGP only sends the best route

Going forward


Already run BGP to some remote sites


Migrate the core to bgp first


Do a dress rehearsal


Will be a big scary change so plan well


Examine tools


May not be able to assume we will get traps


May have to watch the BGP tables for
changes


Get a test connection in place