GSN Infrastructure Working Group

equableunalaskaSecurity

Dec 9, 2013 (3 years and 8 months ago)

87 views

G
SN Infrastructure Working Group



















GSN Test Plan and

Initial HW
S
etup

















v1

23
-
04
-
2010


v2

04
-
05
-
2010


v3

19
-
05
-
2010


v4

28
-
05
-
2010


v5

02
-
06
-
2010


v6 13
-
07
-
2010


v7

03
-
08
-
2010


v8

03
-
09
-
2010


v9 28
-
10
-
2010


Tab
le of Contents


Network Setup

................................
................................
................................
....................

3

IP Address Allocation Scheme

................................
................................
.......................

3

IP Addr
ess Assignments

................................
................................
................................
.

4

Configuring the Allied Telesis AT
-
8000GS

................................
................................
...

6

Setting up the Arista Network L2 Switch at ÉTS

................................
...........................

8

Configuration of the GSN Servers

................................
................................
......................

9

Install KVM

................................
................................
................................
....................

9

Mounting the Storage Array Locate
d at ÉTS

................................
................................
.

9

Status Update

................................
................................
................................
....................

10

Appendix A

................................
................................
................................
.......................

13

Configuring the Rarita
n PDU

................................
................................
.......................

13

Creating User Group and User Names for the GSN Client UI

................................
.....

16

Appendix B

................................
................................
................................
.......................

17

Configuration of the IOLAN DS1 Serial
-
to
-
IP Converter

................................
............

17

Network
Setup

IP Address

A
llocation
S
cheme


We will use the following private IP Addressing scheme

which was proposed
early on in
the project
. We
have confirmed with most sites that this IP range will not conflict with
their local network.


10.20.100
.0/24


Due to the number of unknown devices at this point in time at each site, we are proposing
to assign a
fixed
s
et

of addresses to each site. Bec
ause
É
TS is the hub, and has the data
storage and multiple servers, we have allocated more address
es

to them.


10.20.100.1



10.20.100.20

É
TS

10.20.100.21



10.20.100.30

CRC

10.20
.100.31



10.20.100.40

Bastionh
ost

10.20.100.41



10.20.100.50

Cybera

10.20.1
00.51



10.20.100.60

HEAnet


10.20.
100.61



10.20.100.70

HEAnet 2

10.20.100.71



10.20.100.80

i2CAT

10.20.100.81



10.20.100.90

IBBT

10.20.100.91



10.20.100.100


10.20.100.101



10.20.100.110

10.20.100.111



10.20.100.120


10.20.100.121

and up will be use
d for the VMs using a DHCP server.
The
DHCP server
is

located at
É
TS.

The GSN IP addressing scheme is depicted in Figure 1.






Figure 1.
GSN
IP Addressing Scheme


IP Address Assignments


ÉTS:

10.20.100.1



10.20.100.2

ETS1

10.20.100.3

ETS2

10
.20.100.4

ETS3

10.20.100.5

ServerTech PDU

10.20.100.6

Raritan PDU

10.20.100.7

Storage1

10.20.100.8

Storage2

10.20.100.9

Arista Management

10.20.100.10 idp
-
shibb (VM
shibboleth
)

security

10.20.100.18 Synchromedia PC 1 (SmartBoard)



10.20.100.19


Synchromedia PC 2 (Barco)


10.20.100.251

Internet

G
ateway

10.20.100.252

OpenVPN (upcoming)

207.162.8.16 Arista Network L2/L3 switch

207.162.8.17


Allied Telesis L2 switch



CRC
:

10.20.100.21


Raritan
PDU

10.20.100.22


BADLAB PC

10.20.100.23


Dell R710 Server

10.20.100.24


Iperf Server

10.20.100.26

D
evelopment

Server (Window
s)

10.20.100.27
Development Server (Linux)

10.20.100.28


GreenMonitor VM

?

10.20.100.29



142.92.72.100
Allied Telesis L3 Remote management


142.92.72.102
IOLAN DS1 Serial to Ethernet Device (Outback Mate access)
?

142.92.72.103
DellServer eth
1

142.92.72.104
Wattsup?.net

142.92.71.150

VM for Wattsup?.net meter


Bastionhost:

10.20.100.31
Allied Telesis

10.20.100.32
HP Server


Cybera:

10.20.100.41
Dell R710 Server

10.20.100.42
Dell R710 Server

10.20.100.43
Raritan PDU

10.20.100.44
IOLAN Serial to Ethernet Device (Outback Mate ac
cess)

206.95.91.25
Allied Telesis

Management


HEAnet:

10.20.100.51
HEAnet

10.20.100.52
R710 Dell Server eth0

10.20.100.53

10.20.100.54
R710 Dell Server eth1

10.20.100.55
Cisco 2950 SW1

10.20.100.56
Cis
co 2950 SW2

10.20.100.57
Advocent PDU #1

10.20.100.58
Advocent
PDU #2

10.20.100.59

10.20.100.60

10.20.100.61


RackForce
:

10.20.100.81

Virtual Interface on L2 device


A d
iagram of the GSN network with all the STS circuits and p
ort assignments is shown
in
Figure 2
.



Figure 2.
CANARIE Network Resources Allocated to the

GSN


Configuring the Allied Telesis AT
-
8000GS


1)

Set

up and configure the Allied Telesis AT
-
8000GS
. For the CLI Commands of
Allied Telesis, please refer to,


http://www.alliedtelesis.com/media/datasheets/guides/AT
-
S95_V20019_CLI_Guide_RevA.pdf


a.

Configure startup from terminal window


Using the RS232 adaptor that co
mes with the Allied Telesis, you can
connect to the switching using a terminal emulator application like
HyperTerminal from Microsoft Windows.


-

Set the data format to 8 data bits, 1 stop bit, and no parity.

-

Set Flow Control to
none
.

-

Under
Properties
,

select
VT100 for Emulation
mode.




The default user name is “manager” and default password is “friend”.


Configure enable password, hostname, username, remote access, snmp,
etc. CLI is very similar to Cisco’s CLI.


# configure

(config)# hostname <hostn
ame>

(config)# username <username> password <password>

(config)# snmp
-
server community public ro


(config)# exit


# clock hh:mm:ss may 3 2010




b.

Configure interface for GreenStar Network


# configure



(config)# Vlan database

(config
-
vlan)# vlan 100

(confi
g
-
vlan)# exit

(config)# exit

#


# configure

(config)# interface vlan 100

(config
-
if)#
description

GreenStar

Network

(config
-
if)# ip address 10.20.100.x 255.255.255.0

(config
-
if)# exit

(config)# interface ethernet
g25

(config
-
if)# switchport mode access

(co
nfig
-
if)# switchport access vlan 100

(config
-
if)# no shutdown

(config
-
if)# exit

(config)# exit

#


# copy running
-
config startup
-
config // This will save the configuration


2)

Establish an L3 connection to the Allied Telesis AT
-
8000GS for remote access.

No
te: This does not require a GbE connection. This is only a control plane for
ARGIA access.

3)

Establish a connection from the Allied Telesis AT
-
8000GS (GbE SFP) to the OME
on CANARIE’s network.

4)

Request has been made to CANARIE for the LP from each node to É
TS (and
between RF and GRC). Need to obtain the resource from CANARIE to have the
circuits all in place.

a.

CANARIE has indicated that the circuits will most likely be STS
-
3c
-
7v. May
be able to request STS
-
24c.

5)

Establish a ping between the AT
-
8000GS interf
aces once the circuits on the
CANARIE Network are in place.

6)

Throughput testing: Would request a simple PC with Iperf running on the GSN APN
for throughput testing to verify that we are achieving Gbps rates.

a.

Will require SSH access or have Iperf daemon run
ning.

7)

Install KVM on the Dell server as the host OS.

8)

Connect the Dell server (when ready) to the AT
-
8000GS switch.


Setting up the Arista Network L2 Switch at ÉTS


The Arista Network L2 switch will be used for routing the tagged VLANs coming from
the exten
ded nodes coming from HEAnet in Ireland, IBBT in Belgium, and i2CAT in
Spain.


Current IP address for the 7124S at ÉTS is 217.162.8.16.


# configure terminal

(config)# hostname arista
-
gsn

(config)# interface management 1

(config
-
if
-
Ma1)# ip address 10.20
.100.9 255.255.255.0

(config
-
if
-
Ma1)# interface management 2

(config
-
if
-
Ma2)# ip address 217.162.9.16 255.255.255.0

(config
-
if
-
Ma2)# exit

(config)# ip routing

(config)#
snmp
-
server community public RO

(config)# vlan 100

(config)# vlan 153

#HEAnet tagged VL
AN ID

(config)# vlan 160

#HEAnet tagged VLAN ID

(config)# vlan 960

#HEAnet tagged VLAN ID

(config)# interface ethernet 21

(config
-
if
-
Et21)#

description "1490nm
-

BastionHost"

(config
-
if
-
Et21)#

switchport access vlan 100

(config
-
if
-
Et21)#

interface ethernet

22

(config
-
if
-
Et22)# description "1530nm
-

Cybera"

(config
-
if
-
Et22)#
switchport access vlan 100

(config
-
if
-
Et22)# interface Ethernet23

(config
-
if
-
Et23)# description "1550nm
-

CRC"

(config
-
if
-
Et23)#
switchport access vlan 100

(config
-
if
-
Et23)# interface Et
hernet24

(config
-
if
-
Et24)# description "1570nm
-

HEAnet"

(config
-
if
-
Et24)# switchport mode trunk

(config
-
if
-
Et24)# switchport truck allow vlan all



Configuration of the GSN Servers


Before booting up and installing KVM, the
Intel's VT
-
x feature must be en
abled in the
BIOS for virtualization support. As well, the server BIOS must be configured to
automatically boot when power is detected. This can be found under the power
management option within the BIOS.


Install KVM

1.

Install the Operating System. Ubuntu

10.04
LTS 64
-
bit OS

2.

Activate the virtualization capability of PCs. This function is found in the
"Performance" category of the setup menu of PCs (press F2 or F12 when booting a
PC).

3.

Install kvm: sudo apt
-
get install kvm

4.

Install libvrt: sudo apt
-
get instal
l libvirt
-
bin

5.

Once installed, you can verify that everything okay by running:

kvm
-
ok

If you get the following message you are good to go, otherwise something not
configured or the BIOS is not set for HW Virtualization:

# kvm
-
ok

INFO: Your CPU supports KVM
extensions

INFO: /dev/kvm exists

KVM acceleration can be used

Mounting the
S
torage
A
rray
L
ocated at ÉTS

1.

Mount a shared storage to all the hosts. I use sshfs:


sshfs
-
o idmap=user
$USER@storageIP:/space/r0 /storage



To interconnect the testbed with the main

GSN testbed, you will need to use the
same shared storage at ÉTS.




There are two storage servers: Storage 1 (10.20.100.7) and Storage 2
(10.20.100.8)



The mounting point for Storage 1 should be: /storage in all GSN servers. The
storage capability is 1.7 T
B



Storage 2 is used as backup for Storage 1. In other words, VMs data will be saved
in Storage 1 and the whole volume of data will be backup daily. If the Storage 1
fails, the Storage 2 will be mounted at exactly the same mounting point /storage.



The mount

command must be put in /etc/fstab

2.

If you want to move a VM from host1 to host2, make sure that both hosts appear
in the /etc/hosts of each other, because by default libvirt does not recognize IP
addresses.

3.

You may want to install virt
-
manager in order to
have a graphical interface. The
current version of virt
-
manager has some bugs when moving VMs. So it would be
better to install virt
-
manager 0.8.4:


http://pyl.pylanglois.com/2010/05/12/how
-
i
-
installed
-
virt
-
manager
-
0
-
8
-
4
-
on
-
ubuntu
-
10
-
04/

4.

Use vmbuilder to create vm:



sudo apt
-
get install ubuntu
-
vm
-
builder



cd /home/vmm



sudo ubuntu
-
vm
-
builder kvm karmic
--
addpkg openssh
-
server
--
addpkg screen
--
mem

256
--
libvirt qemu:///system

5.

Start VM and move it using virt
-
manager or virsh command line:



sudo virsh;



virsh#start ubuntu



virsh#migrate ubuntu qemu+ssh://remotehost/system

Status Update


April 30
th
:


LP between CRC and
É
TS established.

Ping resolved bet
ween CRC and
É
TS
.


May 3
rd
:

Remote

access to the
É
TS switch from CRC over the LP.

May 4
th
:

Configuration of Ubuntu 10.04 LTS Server on R710 at CRC.

May 4
th
:

Fibre pull
ed

between the BADLAB and roof of building 2A.


May 13
th
:

Confirmed connectivity to

various devices at ÉTS from CRC.

See ÉTS IP
Address
es

below to see wh
ich

addresses are

ping
-
able.

May 13
th
:

Slot and channel assignment over the CANARIE network assigned by
Thomas Tam. See diagram below.

May 14
th
:

Virtual interface at RackForce was as
signed

May 17
th
:

Circuit to HEAnet terminated on the L2SS card and then connected to the
ÉTS
Allied Telesis.



Allied Telesis port assignment at ÉTS

port g21 = 1490 nm channel (Bastion Host)


Link status:
Not connected yet

port g22 = 1550 nm channel

(CRC
)


Link status: up

port g23 = 1530 nm channel
(Cybera)


Link status: up (temporarily to RF)

port g24 = 1570 nm channel

(HEAnet)


Link status: up

May 19
th
:


Initiated
L2 connectivity
tests
between ÉTS and RF and between ÉTS and
HEAnet.

May 20
th
:

Fibre
that was laid out between the BADLAB and the enclosure terminated.

May 21
st
:

Test HEAnet VLAN 160: Configured the interface port facing HEAnet to
tagged, however, cannot ping or see any MAC addresses from HEAnet.
CANARIE redirected the LP from HEAnet to

terminate directly to CRC
(bypassing the L2SS card). Unable to ping or see MAC address from HEAnet
from CRC. RackForce has configured a virtual interface on a device at their
end with an IP address of 10.20.100.81. Tested the circuit from
ÉTS
to
RackFo
rce and was unable to resolve. Confirmed with RackForce that the
circuit at their end is configured to untagged. Debugging is in process.

May 25
th
:

BastionHost has configured and powered up the Allied Telesis AT
-
8000GS
switch. Currently waiting for the
SX optics for the Allied Telesis, this will
terminate on the TSM R400 module from EastLink.

May 26
th
:

Removed HEAnet VLAN 160 and tested VLAN 153 to CRC. Ping was
successful and MAC address from HEAnet was seen. Reestablished VLAN
153 to ÉTS through th
e L2SS card, ping was unsuccessful and no MAC
address updated. CANARIE will look into configuration options on the L2SS
card. We have contacted Dante to find out if there are any issues with the
VLAN 160 circuit

May 28
th
: Reconfigured the circuit betwee
n ÉTS and CRC from untagged to tagged.

June 1
st
:

Cybera received the R710 from Dell. They are now in the process of
installing Ubuntu Server 10.04 LTS 64
-
bit OS and KVM software.

June 2
nd
:

Established a connection from CRC over the trunked line t
o
ÉTS to HEAnet
over the L2SS card.

From CRC, I am able to ping HEAnet successfully over
VLAN 153. Problem was on the L2SS card. The connection from the GE
port facing the Allied Telesis at
ÉTS

was not configured to the right WAN
port on the L2SS card.


June 2
nd
:

All three tagged circuits from HEAnet (VLAN ID 153, 160 and 960) are now
terminated onto the L2SS card, Gigabit Ethernet port facing
É
TS.

June 2
nd
:

Cybera’s Dell R710 server configured with Ubuntu 10.04 64
-
bit OS. SSH
server installed for remo
te access.

June 3
rd
:

Ran throughput tests between CRC and HEAnet over VLAN ID 153. Tests
involved the LP from CRC to
É
TS and from
É
TS to HEAnet. VLAN ID 153
is a 155 Mbps circuit. UDP traffic was generated using iPerf. Was achieving
full 155 Mbps throu
ghput. TCP traffic was also generated; however we were
only achieving 15
-
20 Mbps. Tweaked out a number of parameters within the
TCP stacks and as time progressed, max throughput achieved was 75
-
80
Mbps.

June10
th
:

Test and configure the second tagged circ
uit from CRC to HEAnet via
É
TS.
Ping test between HEAnet to CRC’s server was established.

June 10
th
:

L2 switch at Cybera is configured. No fibre to the roof where the GSN
equipment is to be installed.

June 16
th
:

On May 25
th
, the L2 switch at Bastion H
ost was terminating the EastLink
connection on a temporary Gigabit Ethernet port (Copper). Now the EastLink
connection has been terminated on the SFP port g21. A temporary windows
box has been installed and connected to port g11. CANARIE is still waitin
g
for ACORN to finish the fibre ring and terminate the circuit onto CANARIE’s
OME in Halifax. The circuit from Bastion Host is a tagged circuit with
VLAN ID 1126. CANARIE has made a request on CRC’s behalf to ACRON
to see if it is possible to untag the c
ircuit on the L2 switch at ACORN right
before CANARIE’s OME 6500. ETA


end of the month.

June 18
th
:

Made a request to CANARIE to terminate the RackForce circuit to CRC for
testing purposes.


June 22
nd
:

RackForce Circuit terminated to CRC bypassing
the É
TS switch. Configured
the circuit to untagged and was unable to ping or see any MAC addresses
from RackForce.

June 24
th
:

A temporary circuit from EastLink was terminated onto CANARIE’s OME to
connect Bastion Host to
É
TS. Circuit is terminated on the CWDM

gear on
channel 1490nm in Montreal. Not able to ping over the LP.

June 25
th
:

Connection from RackForce to CRC was reachable. Email from Keelan at
RackForce indicated that the service provider terminated the fibre on the
incorrect ports. Once this was

c
orrected
, ping was achieved from both ends

and MAC addresses were seen from both ends
.

June 25
th
:

CRC made the request to CANARIE

to

move the circuit from Bastionh
os
t to
CRC for testing purposes (s
imilar to the connection from RackForce
)
.

June 28
th
:

Con
nection from Bastion Host to CRC was configured and tested. Ping was
achieved

from CRC to Bastion Host. Tags are being removed from the
packets on the L2 switch at

Eas
t
link before
terminating onto the CANARIE
OME 6500 in Halifax.

June 29
th
:

Circuit from
Bastionh
ost moved back to
ÉTS.

Ping
was
not reachable once
again. Possible issues with the AT
-
8000GS
.

July 6
th
:

Connection from CRC’s server to the remote data storage at
É
TS over the LP.
Was able to SSHFS to the data storage, mount and create a VM of U
buntu.

July

19
th
:

Allied Telesis at Cybera configured with L3 routed IP address. Access to the
switch from CRC was confirmed.

July 21
st
:

Termination of Cybera circuit to
É
TS. Rackforce to GRC was terminated,
however, the transceiver that was supplied by

Cybera was not supported by
the HP Procurve switch.

July 22
nd
:

GRC to procure the supported transceiver for the HP Procurve.

Aug 3
rd
:

Email confirming the LP between RackForce and GRC has been established.

Aug 5
th
:

Configuration of the R710 server at Cy
bera onto the GSN APN network. Ping
was not
achieved
.

Aug 6
th
:

Technical support with Arista Network Engineers to allow 3
rd

party SFPs to
interoperate with the 7124S (ÉTS) and the 71
2
0
T
-
4S (CRC)

Aug
20
th
:

Tested 3
rd

party SFPs on the 71
2
0
T
-
4S at CRC and
then port the command set
to the 7124S at ÉTS. CRC test was successful.

Aug 27
th
:

Ping from CRC to Cybera


Aug

31
st
:

The serial connector is connected directly to the R710 server at Cybera. From
CRC, we were able to use “minicom” as a terminal window to
connect to the
serial port on the Raritan. Reconfigured the Cybera Raritan to 10.20.100.43
netmask 255.255.255.0.
See Appendix A.

Aug 31
st
:

All core GSN nodes are pingable (ÉTS, CRC, Cybera, Bastionhost)

Sept 3
rd
:

At ÉTS, the Allied Telesis CWDM SFPs wer
e removed from the AT
-
8000GS
L2 switch and moved to the Arista 7124S switch. CDWM SFPs were
recognized by the 7124S and ping was resolved between all the core GSN
nodes.




Appendix A


Configuring the Raritan PDU


To configure the Raritan PDU, you will r
equire the serial adaptor that is provided when
you purchase the Raritan PDU. Using a terminal emulator application like
HyperTerminal from Microsoft Windows, you can connect to the Raritan PDU to
configure the IP address for remote access and monitoring
of the PDU. Set the terminal
setting to the following:





When you are connected to the Raritan PDU via the serial interface, you will see the
following CLI commands.






Type “config”. This will allow you to configure the networking parameters fo
r the PDU.




Enter “none” to manually configure the networking parameters. Enable IP Access
Control to “yes” to allow access via the web, telnet or SSH. Save configuration will
upload the new parameters to the Raritan PDU. Connect the LAN port on the
PDU to the
network switch to allow remote access of the PDU.






The picture above is a screen shot showing the HTTP interface of the Raritan PDU.


Alternative option:


In the case of
Cybera, the network configuration of the Raritan PDU was done remo
tely.
If the Raritan unit is installed in the outdoor enclosure and you would like to configure or
make network configuration changes to the Raritan, you can do so by connecting the
serial adapt
e
r to the serial port on the Dell R710 server co
-
located insi
de the enclosure.
Using SSH, log onto the server that should already have Ubuntu 10.04 64
-
bit Server OS
running. Using a terminal emulator like minicom, changes to the network preferences
can be made.


#apt
-
get update

#apt
-
get install minicom

#minicom

s


Th
is will bring you to the minicom

configuration page as seen below:



Go to Serial port setup:




Type A to configure the serial port to /dev/ttyS1 or /dev/ttyS2

Type E to configure the port settings to 9600 8N1

Type F to configure the Hardware Flow
Control to “no”

Type G to configure the Software Flow Control to “yes”

Type “enter” to return to the previous page. ESC will cancel all changes.

Save setup as dfl and this will initial
ize

the serial port modem.

Follow the configuration steps above.


To ex
it minicom: control+a then z and then x to exit.


Creating
U
ser
G
roup and
User

N
ame
s

for the GSN Client UI


Once the

Raritan
PDU has been
configured, a n
ew user group and user names will be
required for remote access

from the middleware team. From the us
er management tab,
select “users and groups”. Under “Group Management”, create a new group called “GSN
users”. After this refreshes, create user “system” under “User Management”. Request
the user change the password
at

the next login. A second user is

also required for the
GSN Middleware team called “synchromedia”.




Appendix B


Configuration of
the
IOLAN

DS1 Serial
-
to
-
IP Converter


Configuring the Perle IOLAN DS1 will require the CD which is included along with the
unit. Connect the unit and the PC

to the same VLAN or using a cross over cable. When
you start the CD, you should see the following screen.




Select EasyConfig to configure the IOLAN with a static IP address (can also use DHCP
if required).




Select IOLAN Easy Configuration Wizard.
This will install an application onto your
local drive.
The a
pplica
tion will detect IOLAN DS1 unit
.




Select “Next” to scan the network for available IOLAN DS1 devices.




All available devices will be listed under the IOLAN List. Select the device yo
u want to
configure and select “Next”.




Assign a System Name and select “Use the following IP address”. Manually enter the IP
address, subnet mask and the default gate
way, and then click “Next”. This

will download
the new configuration to the IOLAN DS
1 device. When
the
download is completed,
connect the Ethernet cable to the correct network port and the DS1 should now be
accessible via a web browser as shown below.