Study on existing Last Mile solutions for circuit emulation applied to research environments

blackstartNetworking and Communications

Oct 26, 2013 (3 years and 8 months ago)

130 views








Study on existing Last Mile
solutions for circuit
emulation
applied to research environments

Version 3.1

November 2010


Jordi Jofre
1
, Joan A. García
-
Espín
1
,

Ana M. Medina
2,

Ronald van der Pol
3
, Pieter de Boer
3,

Igor Idziejczak
3
, Sander Boele
3



1

Net
work Technologies Cluster, i2CAT Foundation (Barcelona, Spain)

2

Network Area, RedIRIS (Madrid, Spain)

3
SARA Computing and Networking Services (Amsterdam, Netherlands)



Table of Contents

1.

Introduction

................................
................................
.....

4

2.

Lambda Station

................................
................................

5

2.1.

Description

................................
................................
.....................

5

2.2.

Working principles

................................
................................
.........

5

2.3.

Promoters

................................
................................
......................

8

2.4.

Last Activities

................................
................................
.................

8

2.
5.

Users

................................
................................
..............................

8

2.6.

Conclusions

................................
................................
....................

8

3.

TeraPaths

................................
................................
.........

9

3.1.

Description

................................
................................
.....................

9

3.2.

Working principles

................................
................................
.........

9

3.3.

Promoters

................................
................................
....................

12

3.4.

Last Activities

................................
................................
...............

12

3.5.

Users

................................
................................
............................

12

3.6.

Conclusions

................................
................................
..................

13

4.

Phoebus

................................
................................
.........

13

4.1.

Description

................................
................................
...................

13

4.2.

Working principles

................................
................................
.......

14

4.3.

Promoters

................................
................................
....................

15

4.4.

Last Activities

................................
................................
...............

16

4.5.

Users

................................
................................
............................

16

4.6.

Conclusions

................................
................................
..................

16

5.

Virtual Routing and Forwarding (VRF)

.............................

16

5.1.

Description

................................
................................
...................

16

5.2.

Working princ
iples

................................
................................
.......

18


5.3.

Promoters

................................
................................
....................

20

5.4.

Last Activities

................................
................................
...............

20

5.5.

Users

................................
................................
............................

20

5.6.

Conclusions

................................
................................
..................

20

6.

Generalizaed Token Based Networking (gTBN)

...............

21

6.1.

Token Based Networking concepts

................................
..............

21

6.2.

Previous work on Token Based Networking

................................
.

21

6.3.

Last Mile Token Based Network
ing

................................
..............

23

6.3.1.

Architecture

................................
................................
..............................

23

6.3.1.1.

Token Builder (TB)

................................
................................
.......................

24

6.3
.1.2.

Token Based Switch (TBS
-
IP)

................................
................................
.......

25

6.3.1.3.

AAA server

................................
................................
................................
...

26

6.3.1.4.

Use Case: User submits a request

................................
...............................

26

6.3.2.

Integration with AutoBAHN

................................
................................
......

27

6.4.

Promoters

................................
................................
....................

28

6.5.

Last Activities

................................
................................
...............

28

6.6.

Users

................................
................................
............................

29

6.7.

Conclusions

................................
................................
..................

29

7.

Comparison Table

................................
..........................

30



1.

Introduction

Connectionless networks, such as the ones based on TCP/IP protocol stack, are present in most
of the premises where scientists run their experiments nowadays (distributed research
institutes laboratories or university campuse
s, among others). Almost all applications requiring
network connectivity rely on packet networks for inter
-
process communication. Cutting edge
research activities need to ensure their applications get the precise data in the precise
moment, leading to expe
riment failure otherwise, which in many cases is direct synonym of
time and money losses.

Since permanent connections are expensive and not as scalable as desired, dynamic on
-
demand circuit emulation in the access network is foreseen to provide a solution
. Therefore,
there is the necessity to reduce the gap between end
-
user applications and bandwidth on
-
demand provisioning systems that operate in the core networks to be able to bring dynamic
circuits until end
-
user sites.

Hybrid networking concepts within

networks as SURFnet, Internet2, Dynamic Circuit Network
(DCN), CA*Net, UCLP, G
-
Lambda and GEANT AutoBAHN allow applications to reserve and use
circuits on demand. Within these networks it is however unclear how to bring those dynamic
circuits until end
-
us
er hosts.

This study is born from a necessity to reduce the gap between end
-
user applications and
bandwidth on
-
demand provisioning systems (last mile network section) that operate in the
core networks. A token
-
based networking architecture is described for

overcoming the last
mile issue. The document revisits existing work done in PHOSPORUS project and proposes a
new definition in the context of a Last Mile solution.

This study is born from a necessity to reduce the gap between end
-
user applications and
ban
dwidth on
-
demand provisioning systems that operate in the core networks. Industry
solutions are considered as objects for comparison, but do not fit the needs of GÉANT3 project
and thus are discarded.

Scientific
-
oriented solutions for overcoming the last m
ile issue (or
“first mile”, seen from user’s perspective) have focused on:



TCP/IP protocol stack enhancement



Low layer circuit provisioning, mainly in layer 2



High performance network processors at the edges

Considering these areas, the set of solutions un
der consideration in this study are:




Lambda Station



Terapaths



Phoebus



Virtual Routing



generalized
Token
-
Based Networking




2.

Lambda Station

2.1.

Description

The upcoming era of LHC high data volume distributed GRID computing implies extra
requirements on the reso
urce management and network aware applications. There is a large
demand for applications that are capable to dynamically allocate a high performance network
path with predictable performance characteristics. Over the past several years, there has been
a gr
eat deal of research effort and funding put into the deployment of optical
-
based, advanced
research networks, such as National Lambda Rail, DataTag, CAnet4, Netherlight, UKLight, and
most recently, the DOE Science UltraNet.

The Lambda Station project is ai
med to enable dynamic allocation of alternate network paths
for traffic of production SciDAC applications and to forward designated flows across LAN,
negotiates with reservation and provisioning systems of WAN control planes. It creates End
-
To
-
End circuit
between single hosts, computer farms or networks with predictable
performance characteristics. Lambda Station project also explores Network Awareness
capabilities.

2.2.

Working principles

Lambda Station is a service that addresses the need for dynamic reconfigu
ration of the local
network infrastructure to enable use of high bandwidth, alternate network paths by specific
applications for designated traffic flows. The next
Figure
1

illustrates the main idea of the
project:


Figure
1
. Lambda Station architecture

How it works: local applications requests high bandwidth WAN paths that aren’t available over
the normal production WAN network path. Having received such a request, Lambda Station

would neg
otiate the availability of the alternate WAN path, and if available, coordinate its
setup and teardown. Since these advanced research networks are typically restricted
-
use
resources, Lambda Station could schedule their use, if necessary. Finally, Lambda S
tation
would coordinate dynamic reconfiguration of the local network infrastructure to selectively
forward the application’s traffic over the alternate WAN path.

The Lambda Station (LS) is a host running special software capable of interact to the storage
system, local network infrastructure and advanced WAN. This SW consists in five modules:



Local Demand module: The interface to host systems or resource manager. Accepts
alternate WAN paths requests, coordinates flow identification keys (i.e.,
source/destin
ation addresses and ports) and provides state information on alternate
path to requesting end system (setup status, bandwidth, etc.).



Local Steering Service: Customizable interface to local network infrastructure. Modifies
local forwarding policies for eac
h specific flow.



WAN module: Customizable interface to alternate path networks. It checks alternate
path availability and negotiates alternate WAN path setup/teardown.



Peer Lambda Station module: Coordinates with peer Lambda Station at remote site to
esta
blish path symmetry. Exchanges flow identification selectors. Acknowledges WAN
setup/teardown, and LAN forwarding reconfiguration. Verifies alternate WAN path
continuity. Provides notification of WAN path or traffic termination.



Authentication/Authorizati
on module: Provide AA for alternate path requests.



Figure
2
.
Components of Lambda Station

Lambda Station must be able to forward traffic on a per
-
flow basis. That’s why LS identifies
PBR clients, which means entities that genera
te flows of traffic that could be subjected to
policy based routing (PBR). So, each PBR client has a group of attributes: source or destination

addresses, ports, ranges, etc. Hence, LS is capable of dynamically reconfiguring local network
devices based on
these attributes.

Initially, flow identification is based on DSCP (differentiated services code point) and Lambda
Station software does support two different modes to work with DSCP:



In the first mode, a site may choose to use fixed DSCP values to identif
y all traffic that
will be switched by LS. Then, Lambda Station advises applications when to apply that
DSCP value, and router configurations remain constant. This mode will typically be
used by sites that do not want their network devices dynamically reco
nfigured under
Lambda Station's control.



In the second mode, a DSCP value is assigned on a per ticket basis by the local LS. The
same DSCP code can be used by multiple tickets as long as the source and/or
destination IP addresses are used as additional flo
w selectors.

Once, LS has identified the flow (PBR client), needs to reconfigure network devices with a set
of rules (PBR rules). Configuring these rules involves creating route map statements, applying
them to appropriate interfaces and creating access co
ntrol list to match traffic. The changes
must be applied for one or both directions of traffic. At the moment, LS uses statically pre
-
configured route map statements. However, PBR rules should be created dynamically on
demand of applications. At this point
, Lambda Station deals with the last
-
mile problem
.

Typically, a campus network can be presented by a hierarchical design with several logical
layers. It consists of work group, core and border layers. Depending on site's specific network
structure, access
to R&D networks (Research and Dynamic Networks) for different user groups
may need to be configured at several layers. For the architecture in
Figure
3

below, outbound
traffic of
WG
-
B

can be switched to R&D network
s at the work group layer because it has a
direct connection to the R&D admission devices. In order to get incoming traffic from R&D
networks forwarded via a symmetric path, the inbound route for
WG
-
B

needs to be configured
at the R&D layer.
WG
-
A

has no di
rect connection to R&D from its work group layer, so PBR
rules must be applied at the network core and R&D layer for both inbound and outbound
traffic.



Figure
3
.
A hierarchical network model

2.3.

Promoters



U.S. Departament of Energy (DOE)




Fermi National Accelerator Laboratory (Fermilab)



California Institute of Technology (
CALTECH)

2.4.

Last Activities



JointTech/ESCC Winter 208, oral presentation, Honolulu, Hl, January 20
-
25, 2008



CHEP07, oral presentation, Victoria BC, Canada, September 2
-
4, 2007


2.5.

Users



Fermilab Storage Resource Manager, Tier1 SRM



Center and Computing for LHC Physics in the US, CMS Tier1

2.6.

Conclusions

The initial release of Lambda Station Software (1.0) is a fully functional prototype, including
interact
ions between: applications and Lambda Station
s, Lambda S
tation

and the site LAN and
pairs of Lambda S
tations, but no interoperability across different platform.

This software uses multiple API
-
calls or primitives, but we focus this section describing an
operational function:
openServiceTicket
, used
by
applications and remote Lambda S
tations to
request an alternative network path.

In the simplest case, an application or a remote Lambda Station places an
openServiceTicket

request, and specifies source and destination PBR clients, desired bandwidth, bo
arding (a time
when Lambda Station can begin configuring the network), start and end times for data

movement. A unique ID will be returned immediately in response to an authenticated and
authorized
openServiceTicket

request. This ID can be used by applicat
ions to track the status of
path provisioning, getting additional information needed for flow marking, e.g. DSCP assigned
by remote Lambda Station to the corresponding ticket at its end, as well as to synchronize
actions with the remote site if, for exampl
e, the remote application cancels the ticket.

The way to deal with last
-
mile problem will be to apply the same set of policy rules to a logical
grouping of devices. As long as the same PBR rules are applied on several layers of hierarchical
work group arch
itecture Lambda Station network model can be represented by only one group
of devices.

At this time, a site’s infrastructure must be configured for PBR and described by the network
administrator in the configuration of a site’s Lambda Station. There is a
work in progress to
implement a feature that allows LS to discover a site’s PBR configuration automatically.


3.

TeraPaths

3.1.

Description

TeraPaths is a U.S. Department of Energy (DOE)
-
funded project conceived to address the
needs of the high energy and nuclear
physics scientific community for effectively protecting
data flows of various levels of priority through modern high
-
speed networks. The primary
motivation of the project comes from the world of modern high energy and nuclear physics
(RHIC, LHC, ATLAS, U.S
. ATLAS, CMS), where extremely large quantities of experimental and
analysis data need to be transferred through high
-
speed networks across the globe, to be
shared among scientists participating in various experiments

Despite the fact that not all network
flows are of equal priority and/or importance, the default
behaviour of the network is to treat them as equal priority. Thus, the competition among flows
for network bandwidth can cause severe slowdowns for all flows, independent of their
importance. TeraP
aths offers the capability to distinguish between various data flows and
enable the network to treat each flow differently in a pre
-
determined way through the use of
the differentiated networking services architecture (DiffServ).

The TeraPaths project at B
rookhaven National Laboratory (BNL)

investigates the combination
of DiffServ
-
based LAN QoS with WAN MPLS tunnels in creating end
-
to
-
end (host
-
to
-
host)
virtual paths with bandwidth guarantees. These virtual paths prioritize, protect, and throttle
network fl
ows in accordance with site agreements and user requests, and prevent the
disruptive effects that conventional network flows can cause in one another. TeraPaths
automates the establishment of network paths with QoS guarantees between end sites by
configuri
ng their corresponding LANs and requesting MPLS paths through WANs on behalf of
end users.

3.2.

Working principles


TeraPaths offers the capability to distinguish between various data flows and enable the
network treat each flow differently in a pre
-
determined w
ay through the use of the
differentiated networking services architecture.

The network devices of the LAN of each end site are under control of a TeraPaths system
instance. They follow the approach of conceptually dividing the end
-
to
-
end route between a
so
urce and destination end
-
sites in to three significant segments:

1) The segment within the LAN of the source end
-
site: from the host where the data
reside at the beginning of a transfer to the site’s border router.

2) The segment within the LAN of the dest
ination end
-
site: from this site’s border
router to the host where the data will arrive.

3) The segment connecting the source and destination site border routers (WAN): This
may consist of multiple network segments belonging to different administrative
dom
ains.


Figure
4
.

End
-
to
-
end route segment division

Terapaths follows a hybrid star/daisy chain setup model, where the initiating end
-
site
coordinates with the target site. If a mutual agreement is achieved, the initiating system
c
ontacts the appropriate WAN provider and relying on that provider to coordinate, if
necessary, with other WAN domains along the desired route. The configuration of the path is
successful if all three parties are successful in configuring their segment of t
he path.


Figure
5
.

Star/daisy chain hybrid model

The Terapaths software configures and manages LAN QoS paths from end host to border
routers. Terapaths partitions an end
-
site’s bandwidth into multiple classes of service with
diff
erent priorities with statically or dynamically assigned bandwidth. DiffServ architecture is a

packet
-
oriented approach and the distinction between data packets is done by means of their
Differentiated Service Code Point (DSCP) markings. Packets receive di
fferent treatment
according to their DSCP (utilizes six bits of a header’s ToS IP byte). Traffic needs to be
conditioned (policed/shaped/marked) only at network boundaries. Therefore, the host routers
play the role of control valves, throttling and marking

data flows outgoing from host and
incoming to the site’s network. The rest of the network simply honours the packet markings all
the way out the site and into the WAN.

The TeraPaths system does not have direct control over the network devices of the WAN
route
segment. For TeraPaths, the WAN segment is a “cloud” through which packets travel from
source to destination. The cloud representation is relevant in the sense that the WAN domain
is an independent entity that can be contacted in a number of ways to
make arrangements for
a specific data flow.

End
-
sites are clients to WAN providers and have the capability to request special treatment for
specific data flows to the degree that their provider allows. Furthermore, WAN providers are
likely to adopt servic
e level agreements between them so as to make certain QoS options
available to their clients.

The automated configuration of the WAN providers themselves is more complex, requiring the
existence of suitable software mechanism exposed by the WAN providers a
nd collaboration
between the responsible parties. TeraPaths is
designed to

request guaranteed bandwidth
WAN paths by invoking the web services of a WAN provider. To deal with any number of
different WAN providers web service interface implementations, as w
ell as possible
compatibility issues, TeraPaths uses façade objects in the form of proxy server modules. Each
such module, one per provider, wraps the services the corresponding providers makes
available and exposes a common interface compatible to the cor
e TeraPaths module
requirements. For instance, is here where AutoBAHN should provide an interface if wants to
use TeraPaths.

The level of QoS that is possible within WAN segments affects the level of QoS guarantees that
TeraPaths can provide. Typical case
s are the following:

-
No WAN QoS available
: Prioritization/protection is possible only within end site LANs,
while the flow will be treated as standard “best effort” traffic in the WAN. If the WAN
becomes increasingly loaded, the requested bandwidth may no
t be honoured due to
the lack of QoS

-
WAN QoS statically available
: End
-
to
-
end data flow prioritization/protection is
possible; however, the WAN providers along a route have to agree to configure in
advance their network devices using DiffServ techniques s
imilar to those at the end
sites or MPLS.

-
WAN MPLS tunnel dynamic configuration
: End
-
to
-
end prioritization/protection is
possible on demand. This case assumes that WAN providers have publicly available
services allowing the dynamic configuration of MPLS t
unnels between specific end
-
sites. A user is able to set up a virtual path with flow
-
level granularity



Figure
6
.

End
-
to
-
end path setup

The TeraPaths project demonstrate that the combination of LAN QoS techniques, based on the
Dif
fServ architecture combined with WAN MPLS tunnels, is a feasible and reliable approach to
providing end
-
to
-
end, dedicated bandwidth path to data flows in a demanding, distributed,
computational environment. Furthermore, TeraPaths technology offers a flexib
le way to
partition a site’s available bandwidth into predetermined bandwidth slots, and to protect
various data flows from competing against each other.

3.3.

Promoters



U.S. Department of Energy (DOE)



Brookhaven National Laboratory (BNL)


3.4.

Last Activities



TeraPaths demo in SC07
: International Conference for High Performance Computing,
Networking, Storage and Anal
ysis.



GridNets 2006
.


3.5.

Users



The Relativistic Heavy Ion Collider (RHIC) at BNL



The Large Hadro
n Collider (LHC)



The ATLAS experiment



The USATLAS project



The CMS experiment



The ESnet OSCARS project



Ultralight project



3.6.

Conclusions

TeraPaths solution is compatible with AutoBAHN’s needs. TeraPaths follow the approach of
conceptually dividing the end
-
to
-
end route between a sou
rce and a destination end sites in to
three significant segments:

-

The segment within the LAN of the source end
-
site.

-

The segment connecting the source and destination site border routers (WAN).

-

The segment within the LAN of the destination end
-
site.


For

TeraPaths, the WAN segment is a “cloud” through which packets travel from source to
destination. The cloud representation is relevant in the sense that the WAN domain is an
independent entity. This WAN segment is where AutoBAHN set
-
up the edge
-
to
-
edge
con
nections.

This solution can be deployed implementing a TeraPaths instance in each local network
connected to an NREN and providing an interface between TeraPaths and AutoBAHN.


4.

Phoebus

4.1.

Description

Phoebus is an environment for high
-
performance optical net
works that seamlessly creates
various adaptation points in the network to maximize performance. By splitting the network
path into distinct segments, Phoebus minimizes the impact of packet loss and latency by
leveraging the best performance attributes of e
ach network segment. Using an end
-
to
-
end
session protocol, transport and signalling adaptation points can be controlled and better
performance is possible. The Phoebus adaptation library also allows existing applications to
take advantage of advanced netwo
rks with little or no modification. The Phoebus project aims
to help bridge the performance gap by bringing revolutionary networks like Internet2's
Dynamic Circuit Network (DCN) Infrastructure to users. Many applications can begin to use
DCS (Dynamic Circu
it Services) with no modification by being transparently authenticated and
redirected to the circuit network via a Phoebus service node.

Martin Swany, Assistant Professor, Department of Computer and Information Sciences,
University of Delaware and Internet
2 faculty fellow explained, "Due to the huge success and
growth of the Internet, it has been difficult to deploy disruptive technologies broadly. Phoebus
(developed by researchers at the University of Delaware) builds on standard Internet
infrastructure at

the edge of the network, while creating a bridge to advanced optical and
packet networks like the new Internet2 network, enabling application communities like
bioinformatics and satellite/telescope data to benefit from hybrid networking in the short
term.

In the longer term, Phoebus represents an architectural evolution of the Internet in
which commercial network providers can offer a richer set of services to their users because
they will no longer be constrained by the performance limitations of TCP."



4.2.

W
orking principles

Phoebus works by changing the current Internet model, which binds all end
-
to
-
end
communication to Transport layer protocols such as TCP and UDP. Using an end
-
to
-
end layer
called the Session layer, Phoebus can segment a connection into a s
eries of Transport layer
connections. The edge connections can still use TCP and in many cases, existing software
works with no modification. These edge connections are serviced by Phoebus Gateways (PG).
The flow of traffic over the backbone network is han
dled by a series of these gateways. The
gateways can choose the best protocol and settings for sending the data across the network,
and can allocate additional network capacity on behalf of the user.


Figure 7.
TCP model vs. Phoebus model

So Phoebus Gatew
ays are placed at network access points and can be thought of as network
“on
-
ramps”. These gateways can then intelligently move the data as efficiently as possible to
the other edge of the network. At the other edge, data are received as
regular TCP/IP tra
ffic
,
thus

destination
client
does not
need to be modified. It’s important to note that this method is
not altering the path, just terminating TCP connections and providing short
-
term buffering,
also known as “cascading TCP connections”.

TCP on its own, as

end to end
protocol

cannot mitigate the problem of the edge networks. TCP
is heavily dependent on the RTT (Round
-
Trip Time). RTT is one of the most important
performance parameters in a TCP exchange. It’s defined as the time required for a host
transmits
a TCP packet to its peer, plus the time for an acknowledgment to be received. If the
reply does not come within the expected period, the packet is assumed to have been lost and
the data is retransmitted. RTT time is also known as the
ping

time. It’s a big
factor in helping
how to deal with timeouts and retransmissions and may be used to find the best possible
route, due to a decrease in the RTT can produce an increase in throughput.

In Phoebus context, the RTT of an individual connection is less than the R
TT of the end
-
to
-
end
connection. So, establishing two TCP connections, each having a shorter RTT, the speed of
each of the shorter connections should be faster than the speed of the overall connection. If
this is the case, the throughput of the overall con
nection in the Phoebus case should run close
to the speed of the slowest individual connection.


On the other hand, when a loss occurs on an end
-
to
-
end connection, it causes a reduction in
the throughput for the entire connection. However, with the shorter,

independent
connections, the throughput is only dropped for one portion of the connection. The others
continue running at the same speed. This works to prevent intermittent drops from
substantially affecting the entire connection.

In the Phoebus model, th
e end
-
to
-
end connection is broken into at least three segments: a
short RTT connection between the end user and the edge network PG, the negotiated inter
-
PG
connections and the connection from the far
-
side edge PG to the destination host. Also, in the
Phoe
bus model, once a PG has received the data, it becomes responsible for ensuring that that
data is received at the next hop. These two properties combine to provide for improved end
-
to
-
end performance.

Thus, we can think of it as controlling an end
-
to
-
end s
ession comprised of segment
-
specific
transport protocols and
signalling

technology. That’s the function of Phoebus Session Protocol
(PSP), used to manage a multi
-
layer connection.


Figure 8.
Phoebus Session Protocol on protocols stack

After these previous

concepts, the main point is the software that runs on PG and clients. The
current version of the software is 0.9 and is integrated with
the DC Network Software Suite.
Once, this software is installed in PG, these gateways are able to signal the cross
-
doma
in
circuit infrastructure. In clients, Phoebus can be enabled on Linux systems (Windows support
under investigation) and it not involves modifications to the
users’

workstations.


Figure 9.
Phoebus Gateway communication

4.3.

Promoters



Internet2



University of Delaware, Newark




Recently:
GEANT2

and
ESNET



4.4.

Last Activities



Peformance Update at Winter 2008 Joint Techs Workshop



Supercomputing SC2007


4.5.

Users



GridFTP from
the Globus project



The Large Hadron Collider (LHC)


4.6.

Conclusions

Phoebus model separates the whole end
-
to
-
end connection in segments: source host


PG,
inter
-
PG connections and PG


destination host. The point i
s to make short TCP connections to
get good throughput. That’s how PSP (Phoebus Session Protocol) works. This new protocol
requires that applications can use these new features.

Then, user is able to send data via these PGs withous extensive host tuning.
Phoebus solves
last mile issue locating the PGs at the edge of backbone networks and using two methods to
establish connections to them:



Wrapper Library. Overriding the standard operating system socket calls to that any
application linked against it, can t
ransparently use Phoebus infrastructure. The only
requeriment from the users perspective is that he must install the software package
and then set two environmental variables: one to load the library and another to
specify the first PG along the path.



Tran
sparent redirection. Enabling applications to use Phoebus infrastructure available
in Linux. In this scenario, the administrator would setup a machine on his network
running the Phoebus code. The end users would then route their traffic through this
box. T
he administrator could then setup some firewall rules that would transparently
redirect TCP connections destined for specific hosts or services through the forwarding
software. The end user would no longer need to perform any local changes except to
route
their traffic through the forwarding box.


5.

Virtual Routing and Forwarding

(VRF)

5.1.

Description

If the campus network supports VRF, t
his solution requires minimal changes.
VRF

needs to be
enabled onl
y on a router inside the
campus,
which

is in the IP routing p
ath between end host
and the
light path
. No changes are needed on the end hosts. Packets are routed between the

VRF router and the end host via the generic campus infrastructure using normal IP routing.
This solution does not provide explicit QoS. Usually,

this is not a problem inside a campus
network because
over provisioning

in a LAN is easy and cheap. However, there is nothing that
prevents QoS technology to be used in combination with virtual routing.

Light paths

are often used to build Optical Private
Networks (OPNs). Many of these OPNs have
a strict use policy. E.g. only traffic related to the project is allowed to use the
OPN;

all other
traffic must be routed via other paths.

Figure

shows an example of pol
icy requirements:


Figure
1
0
.
Example traffic requirement/flows (this picture does not show all possible
combinations)

In the above example both the red and blue lines represent valid traffic, where red lines use an
OPN, while bl
ue lines do not. As you ca
n see,
all hosts are reachable via the generic routed
internet. The OPN links may only be used if both the source and destination host are part of
that OPN.

The table below shows an overview of the allowed traffic flows between hosts at
site 1 and site
2 i
s

allowed over the OPN A. This is also shown in
Figure


Host related to OPN A

at site 2

Other host at site 2

Host related to OPN A
on site 1

Allowed over OPN A
and general internet (if
available)
1

Not allowed o
ver OPN
A, should use general
internet

Other host on Site 1

Not allowed over OPN
Not allowed over OPN



1

This might be policy related, if the desire is there to have a backup via the general routed

internet this
can be use, but it could also be a policy to explicitly not use the general routed internet.


A, should use general
internet

A, should use general
internet


So,
only traffic between a host related to OPN A on both site 1 and 2 is allowed to use O
PN A.
The host may optionally use the general internet as backup. Any other host at site 1 is not
allowed to use OPN A to reach a host related to OPN A at site 2. These hosts should use the
general internet instead. We could extend this scheme with multipl
e other OPNs where a host
can even be part of multiple OPNs. But the hosts can only use an OPN if both the sending and
the receiving host are part of
it;

otherwise they should use the general internet.

Segregating traffic from different OPNs is a challenge

in itself. Where normal routers do not
care about policies and just focus on the shortest route from A to B, OPNs with use policies
have a requirement to evaluate the source address in the decision for the best route. This
could be solved by simply by buy
ing dedicated routers and configuring static routes, but this is
expensive and does not scale. Another solution is to use VRF by placing interfaces and links of
an OPN into a VRF. VRF is a technology that allows a router to maintain multiple routing tables

or instances. It is even possible to create views where a host or a group of hosts can have
access to multiple OPNs and on the other hand maintain their visibility to/from the general
internet, and thus not becoming exclusively part of an OPN.

5.2.

Working pri
nciples

To illustrate a
VRF

setup, Juniper’s VRF technology will be described. With VRF, multiple
routing table instances are used. Those routing tables are used to send traffic to the
corresponding interface.
Figure
1

shows an example of the policies of various
light path

services.

For example, only OPN related traffic is allowed on the OPN A between Site 1 and Site 2.
Traffic of Site 1 users accessing the Site 2 website should go via the routed internet. Or users

at
Site 3 are not allowed to use the routed internet via Site 1. These policies can be enforced
quite elegantly with VRF.

A router with VRF enabled has several routing table instances. Each routing table instance has
one or more interfaces assigned to it.

The routing table instance is used for traffic on one of
those interfaces. One can think of it as a virtual router with a routing table and a couple of
interfaces. The challenge is to make resources available in multiple VRF’s but maintain the
separation
of routing between different VRFs.

Consider, for example, the following setup where a privileged host on some network is allowed
access to a site through an OPN or dynamic
light path

whereas an unprivileged host is not. The
networks these two hosts belong

to are routed by a VRF enabled router.



Figure
1
1
. Example traffic flow and route distribution diagram

Traffic originating from the privileged host enters the router and immediately hits a routing
policy that states that all route lookups are to be done

in the routing table of VRF shared. This
VRF shared is the glue between the global connectivity the privileged host requires and the
OPN the host needs exclusive access to. This is achieved by importing all OPN routes from VRF
A to VRF shared (3) and by i
nstalling a default route in VRF shared (4) for global connectivity of
the privileged host.

VRF A contains only static routes for traffic destined for the privileged host (2) and dynamic
routes learned via BGP to site B (which can also be static routes). T
his ensures the OPN is kept
private without a default or full routing in its routing table. This means that traffic from site B
can never go anywhere but to the privileged host, simply because there are no routes to other
networks in the routing table of
VRF A. The OPN or dynamic lightpath is thus exclusively used
between site B and the privileged host.

Consider traffic originating from the privileged host that is destined for site B. This traffic hits
the routing policy, so the router does a lookup in the

routing table in VRF shared. It finds a
route via the OPN and takes the shortcut through the OPN to site B (1). Return traffic that is
routed by site B via the OPN is looked up in the routing table of VRF A and is statically routed
to the privileged host
(2).

Traffic from the privileged host destined for other networks than the ones at site B hit a
default route in VRF shared and is routed via the internet (6). Because the privileged host’s
interface is a connected route in the global routing table, traff
ic from the internet can reach
the privileged host without a hitch.


Traffic from the unprivileged host that is connected to the same VRF enabled router is routed
through global routing (5).

The previous example is based on a static lightpath setup. But the

same VRF mechanism can be
used in the case of dynamic lightpaths. A nice example is a cloud by
-
passing setup where traffic
is normally sent over the routed internet. But when a lot of data needs to be transferred, a
dynamic lightpath is set up as a tempor
ary high bandwidth channel. This can be handled by
putting the interface connected to the dynamic service in a dedicated VRF (VRF A in our
example). A static default route pointing to the global routing table is added to this VRF. BGP is
used to receive ro
utes from the dynamic service. As long as the dynamic lightpath is not
established, the default route entry is the only active route entry. Because this entry is
pointing to the global routing table, the traffic is send over the routed internet. When a
dyn
amic lightpath is set up, the BGP peering becomes active and routes from the peer are
received. These routes will be inserted in the VRF Shared and these more specific routes take
precedence over the default route. The result is that the traffic will flow
via the dynamic
lightpath.

5.3.

Promoters



Dutch Tier1 (SARA and Nikhef)

5.4.

Last Activities



VRF is present in most current high
-
end routers.

5.5.

Users



Used in production network of SARA Computing and Network Services for LHCOPN,
DEISA, LOFAR, etc.:
http://www.terena.org/activities/e2e/ws2/slides1/1_SARA_Peter.pdf

and
http://indico.ce
rn.ch/getFile.py/access?contribId=11&resId=0&materialId=slides&conf
Id=50345

5.6.

Conclusions

Using VRF’s to separate traffic and with that complying to use policies of OPN’s, a very flexible
and scalable setup can be build. Using this technology it is possible

to provide access for a host
to multiple OPN’s, while the next host only has access to one OPN and another doesn’t have
access to any. Still complete reach ability of the nodes from the general routed internet is
ensured. Scalability is only limited by co
nfiguration, while for a setup with hardware enforced
separation a new router needs to be bought for every OPN.




6.

Generalizaed Token Based Networking (gTBN)

6.1.

Token Based Networking concepts

Token based networking (TBN) is a new concept for a network that ha
ndles packets according
to a specific tag, built
-
in the packet, called token. TBN uses tokens built from small encrypted
pieces of data. By its mean, a token allows for applying different access / enforcement/
filtering rules/criteria when processing the p
ackets that carry the tokens. For instance, a token
may represent an exclusive right to access a service at a particular for a measured amount of
time.

The word token is an overloaded term. The term is likely to create confusion if we do not
define it in t
he context of our research. A token in this context is defined as: “
A shared abstract
permission that is presented as part of an access request”
. The permission is a small piece of
encrypted data that unambiguously references the specific resource request

context.

A policy enforcement point (
Token based Switch
) authorize the resource usage (e.g., a packet
takes a certain path) only if the built
-
in token (
Token Builder
) matches a cryptographic result
applied to the packet using a set of keys provided before
hand by an authority (
AAA server
).
Tokens can bind to different semantics (in particular, it can be associated with one user, a
group of users, a institute or a grid and share it use when accessing designed resources), and
they decouple the time of authori
sation decision from the time of use (e.g., enable the use of
resource for 10 hours from now or from tomorrow at 5 o'clock). Hence, a token provides a
generic way to match applications to their associated network services.

Moreover, tokens are protocol ind
ependent, which has an important advantage; the token can
be regarded as an aggregation identifier to a network service. Generally, we see four types of
aggregation identifiers that can be combined, as follows:



Identifier to point a service to the NE (mult
icast or transcoding)



Identifier that defines the service consumer (grid application)



Identifier that defines the serviced object (network stream)



Identifier that defines the QoS (security, authorization, robustness, deterministic
property)

6.2.

Previous work o
n Token Based Networking

A
Token Based Networking

architecture is implemented in
UvA

under the
PHOSPORUS

FP6
project. The Token Based Network (TBN) testbed uses Tok
en Based Switch over IP (TBS
-
IP) as a
low
-
level system for traffic switching at high speeds (multi gigabits/sec) based on packet
authentication. TBS
-
IP is fast and safe and uses the latest network processor generation (Intel
IXP2850).

Token Based Switch ov
er IP traffic (TBS
-
IP) is a ForCES router designed and implemented in a
prototype at UvA. Using specific encrypted tokens built
-
in the IP packets, TBS
-
IP allows

connections between a regular connectionless IP (Campus) network and a connection
-
oriented
GMPL
S network of an Optical Network Service provider. The TBS
-
IP router provides a web
-
service configuration/set
-
up interface to a high level authorisation and accounting service
(e.g., AAA servers from UvA, Harmony from UniBonn, OSCARS from Internet2). The TB
S
-
IP
router can be programmed for a number of reserved paths/service levels using an XML
Authorization ticket proposed and implemented in the PHOSPHORUS AAA/AuthZ
infrastructure. Briefly the process is depicted in Figure 1 works as follows:

1)


User ap
plication requests to the UvA
-
TBN gateway router (ForCEG) a path
between the IP addresses (IPsrc, IPdst, startTime, duration, and bandwidth), where
IPsrc must be within its IP campus network, and IPdst should be available within the
GMPLS networks.

2)


The ForCEG service will authenticate the user application and request a path to
the GMPLS authority (Harmony IDB) on behalf of its user applications. As a result, IDB
will generate and return the credentials of using authorised paths in format of a Global

Reservation Identifier (GRI) and a
TokenKey

used of in
-
band encryption.

3)


IDB will prepare all intermediate domains involved in the path provisioning with
the requested credentials (GRI, TokenKey, etc).

4)


Once the start
-
time of the requested p
ath arrives, the received credentials from
IDB are enforced into the TBS
-
IP gateway router at the lowest level (IP packets and
circuits).

5)


When a user application (e.g., vlc) is authorised to connect to an IPdst over
GMPLS network, the
Token Builder

inserts encrypted tokens within the outgoing
packets (media streams) and redirect them towards TBS
-
IP.

6)


TBS
-
IP authenticates every received packet identified by the plain
-
text GRI as a
lookup computation (finds GRI
-
entry into the local AuthZ
-
table)
, then it does:

a.


If the Packet comes from IPcampus network (PKT.IPdst==GRI
-
entry.Port1.IPaddr), then it will re
-
write its IPdst to the correct endpoint
(PKT.IPdst=GRI
-
entry.IPdst) and will push the packet into the proper GMPLS
path, as indicated by

GRI
-
entry.Port2;

b.


If the Packet comes from GMPLS network (PKT.IPdst!=GRI
-
entry.Port2.IPaddr), then it will push the packet into the proper outgoing port
into IPcampus as indicated by GRI
-
entry.Port1;




Figure
13

TBN PHOSPHORUS testbed

The PHOSPORU
S Token Based Switch (TBS
-
IP) prototype implementation shows that a packet
-
based admission control system can be used to dynamically select a fast end
-
to
-
end
connection over hybrid network at gigabit speeds.


6.3.

Last Mile Token Based Networking

Last mile com
prises the network section between the end
-
user site (campus, lab, organization,
etc) and the dynamic circuit access point. Generally, these campus networks work at IP packet
level and do not offer QoS. Hence, this can lead in a poor performance of high
-
d
emanding
applications.

In this chapter, we investigate the implementation of a Token Based Networking as a Last mile
mechanism. For this investigation, we take in consideration the concepts of a Token Based
Networking and the related work done in the field

in PHOSPORUS FP6 project.

6.3.1.

Architecture

In a nutshell, the presented approach performs

authorized path selection

between end
-
user
sites and the campus network edge. A cryptographic hardware acts as a real time path
selection mechanism, using a token
-
key to

recognize tokens inside IP packets. These token
-
keys must be requested, authorized and created first. A client can make this request using web
services mechanism to an AAA server. In Figure 2 is depicted the Last Mile TBN architecture



Figure

14.

Last mi
le Token Based Networking architecture


Last mile TBN architecture consists of three parts:

6.3.1.1.

T
oken Builder (TB)

The Token Builder (TB) is software installed at the end
-
user host to tokenise application traffic.
In our approach, the token is inserted into th
e IP
-
option field. This field can be of variable
length and resides in the IP header. An IP packet containing options should be forwarded
unmodified by a router. A router does not need to understand IP options. This makes the use
of IP options transparent
to non
-
TBS network devices. In PHOSPORUS testbed,
hijackers

are
used for token insertion.

A token created for an IP packet is essentially the result of applying an HMAC algorithm over a
number of packet fields as illustrated in Figure 3. An HMAC algorithm
is a key
-
dependency to
create a one
-
way hash (RFC 2401). For instance, in PHOSPORUS testbed, a HMAC
-
SHA1
algorithm generates the unique token (20 bytes) that will be injected into the packet’s IP
option field. As an IP option field has a header of two byt
es, and network hardware systems
commonly work most efficiently on chunks of four or eight bytes, they reserve 24 bytes for the
IP option field. In order to ensure the token uniqueness for packets, they need to include fields
that are different in every pa
cket. Therefore, some part of the packet data will be included
together with the packet header in the HMAC calculation. We mention that some of the first
64 bytes of an Ethernet frame are masked in order to make the token independent of the IP
header field
s which change when a token is inserted (e.g., header length, total length, IP
checksum) or when the packet crosses intermediate routers (e.g., TTL). The mask also provides
flexibility for packet authentication, so that one could use the (sub)network inste
ad of end
node addresses, or limit the token coverage to the destination IP address only (e.g., by
selectively masking the IP source address).



Figure
15
.
Token creation process for a TBN IPv4.

Note that although in the document we refer to token as an en
tire tag built
-
in the packet, in
our proposal, the tag consist of two parts: a plain
-
text aggregation identifier (GRI) and an
encrypted token.

We may associate a token
-
key with an IP source / destination pair, so all traffic between two
machines is handled

with the same key. However, other forms of aggregation are possible. For
instance, we may handle all traffic between two networks with the same key, or all machines
participating in a Grid experiment, etc.


6.3.1.2.

Token Based Switch (TBS
-
IP)

The Token Based Swit
ch (TBS
-
IP) is a cryptographic hardware able to enforce
selective path
forwarding on per
-
packet basis
. For each received packet, the TBS
-
IP checks whether the
current packet has the appropriate IP option field. On success, a Global Resource Identifier
(GR
I) field, identifying a specific application instance, is extracted. Next, TBS
-
IP uses the GRI
entry to retrieve the token
-
key from the authorization table, already provisioned in the TBS
-
IP
by an AAA authority. Finally, the token
-
key is matched with the p
acket token to authenticate
it.


Figure
16

Authorization table

Currently it is possible to implement such programmable network element using specialized
hardware (network processors, FPGAs or using powerful PCs). For instance, in PHOSPORUS
TBS
-
IP uses the

latest network processor generation (Intel IXP2850) able to switch at multi
-
gigabit speeds.



6.3.1.3.

AAA server

The AAA server is an authority responsible for fetching a user policy from the policy repository
that validates the request and creates the correspondi
ng token
-
key. After the request is
validated and the token key is created, the AAA server distributes the token
-
key and the global
resource identifier (GRI) to the end
-
user and to every TBS
-
IP in the selected path.

6.3.1.4.

Use Case: User submits a request

Suppose
a campus network with several faculty buildings like the one depicted in Figure 5. An
end
-
user located at the Engineering faculty would like to forward his application traffic
through a privileged path within the campus network, which is expected to have
a better
performance than the default IP
-
routing. The procedure to submit such a request is as follows:

1.

End
-
user sends a request to the AAA server authority. AAA server authority
validates the request from a policy repository, if affirmative, generates a t
oken
-
key.

2.

AAA server distributes the token
-
key and the GRI to the end
-
user Token Builder
and to each TBS
-
IP in the selected path. This token
-
key will be used for both
generating tokens at end
-
user site (Token Builder) and for verifying tokens at each
TBS
-
I
P.

3.

End user application starts generating traffic, which is tokenised by the Token
Builder using the previously obtained token
-
key. For each received packet at the
TBS
-
IP equipment, it checks the packet token to check the packet authenticity /
validity/aut
horisation. An authenticated packet is then forwarded to its
corresponding privileged path.

4.

All non
-
authenticated packets are forwarded to the default path.



Figure
17.

User submits a request scenario

Note that, to forward selectively tokenised traffic b
etween end
-
user site and campus network
edge, we must implement a TBS
-
IP in every hop. Otherwise, a non
-
TBS
-
IP router won’t process
the packet token and will take his forwarding decision based on its routing table. Furthermore,
every TBS
-
IP hop must have a
t least two routes to select, one to forward the tokenised traffic
and the second one to forward the default traffic.

6.3.2.

Integration with AutoBAHN

Two end
-
users located at different domains require a connection among them. We suppose
both campus network imple
ment TBS
-
IP capable devices. The dynamic circuit among domains
is requested and provisioned by AutoBAHN. Here we would like to merge both systems, TBS
-
IP
campus network and the dynamic circuit provisioned by AutoBAHN. Such a way, we can
provide to the end
-
users a host
-
to
-
host circuit.

In this scenario, instead of having the AAA server at the campus network, the AAA server
authority has to be implemented as an AutoBAHN service. Hence, there is a single AAA server
for all NREN campus networks, in other words,

an AAA server per domain.

The procedure to establish a host
-
to
-
host circuit works as follows:

1.

End
-
user sends to AutoBAHN a host
-
to
-
host request. Note that, this request has
to indicate that the campus networks implements a TBN last mile system. In such
a

way that the circuit can be extended until the end host and not until the
campus network edge as usual.


2.

AutoBAHN AAA authority validates the credit of the client, if is positive,
AutoBAHN accepts the request. Then AutoBAHN set
-
up the dynamic circuit
among

domains/NRENs.

3.

When the dynamic circuit is successfully reserved, AutoBAHN AAA server
distributes the token
-
key and the GRI to the end
-
user Token Builder and to each
TBS
-
IP in the campus network path. This token
-
key will be used for both
generating token
s at end
-
user site (Token Builder) and for verifying tokens at
each TBS
-
IP.

4.

End user application starts generating traffic, which is tokenised by the Token
Builder using the previously obtained token
-
key. For each received packet at the
TBS
-
IP equipment, i
t checks the packet token to check the packet authenticity /
validity/authorisation. An authenticated packet is then forwarded to its
corresponding campus privileged path. At the campus edge, the tokenised traffic
is forwarding to the dynamic circuit estab
lished by AutoBAHN.

5.

All non
-
authenticated packets are forwarded to the default path.


Figure
18.

Last Mile TBN and AutoBAHN integration

6.4.

Promoters



University of Amsterdam (UvA)


6.5.

Last Activities



TERENA Networking Conference 2009
: gTBN over IPv4 is proposed as well as the
presentation of experiments in a live
-
demo



6.6.

Users



Universi
ty of Amsterdam (UvA)




Phosporus: ForCES Token Based Switch Architecture

(pdf).


6.7.

Conclusions

The main advantage of a Last mile TBN solution is allows bindi
ng application traffic from end
-
sites to established authorized paths within the campus network. Hence, we can guarantee
that no rogue users will use the reserved resources. In other words, TBS architecture offers to
user applications a flexible access con
trol approach/solution to the reserved paths, where the
token represents the right to use a pre
-
established network connection at a specific time.

For its own, TBN does not improve the network performance, it just allows to selective
forward packets accord
ing to a built
-
in token. However, network administrators can apply
traffic engineering. In the sense that, they can intentionally for instance, keep the tokenised
path working in low load to achieve better performance. Or assign as tokenised path the route

with less latency or jitter.

Probably, further detail on how AAA server can be integrated with AutoBAHN to implement
TBN feature in campus networks. Also some more detail about the token
-
key distribution
among domains.

The main drawback is the hardware re
quirements; we do need to install cryptographic
hardware able to enforce selective path forwarding on per packet basis in campus network
infrastructure.




7.

Comparison Table



Lambda Station

TeraPaths

gTBN

Phoebus

VRF

Working principle

Reconfiguration of
c
ampus network
infrastructure.
Selectively forwarding
of designated flows.
Policy based routing
(PBR)

Traffic prioritization.
Uses Differentiated
Service architecture
(DiffServ) for
configuring QoS paths
within the LAN

Selective forwarding of
tokenized tra
ffic. A
cryptographic token is
inserted in every packet
of user application
traffic. Binds
applications to
networks using a token
as a glue

Split end
-
to
-
end
connection into several
segments to improve
performance of each
segment. Phoebus
Session Protocol (
PSP)

Virtual/VPN Routing
and forwarding
enabled on router
inside the campus
(multiple routing tables
per router enabled). No
changes needed on the
end hosts. Using
normal IP routing to
selectively forward
traffic from a host on a
static route out of the
campus. QoS not
explicitly provided but
possible

Software
Requirements

Lambda Station
software at every
campus network

TeraPaths software at
every campus network

Token Builder software
to tokenise traffic at
the end
-
sites. AAA
server to provide/
generate
token keys.

Phoebus Gateway at
the campus network
edge

None


Hardware
Requirements

Routers/switches with
PBR , ACL capability

Routers/switches with
DiffServ capability

Token Based Switch.
Hardware able to
selectively forward
traffic according a
cryptogra
phic built
-
in
token. Needed at each
router/switch device at
the campus network

IP router able to deploy
Phoebus Gateway
technology

Required capabilities
present in most high
-
end routers (VRF in
Juniper, VPN Routing
and forwarding in
Cisco)

Changes needed
on
end hosts

YES

NO

YES

Limited

NO

Middlebox needed

NO

NO

NO

YES

NO

Used in production
environment

YES

YES

NO

NO

YES

Who is behind?

Fermilab and CALTECH

RACF Computing
Facility, a division of
Brookhaven National
Laboratory

University of
Amsterdam (UvA)

Internet2 and
University of Delaware

Dutch Tier1 (SARA and
Nikhef), used in
production network of
SARA Computing and
Network Services for
LHCOPN, DEISA, LOFAR