2 Federal University of Pernambuco - UNIFACS - UFPE

candlewhynotΔιαχείριση Δεδομένων

31 Ιαν 2013 (πριν από 4 χρόνια και 7 μήνες)

653 εμφανίσεις

An Instrumentation and Measurement
Architecture Supporting Multiple Control
Monitoring Frameworks

XXX
Brazilian

Symposium

on

Computer Networks
and

Distributed

Systems

III
Workshop de Pesquisa Experimental da Internet do Futuro (WPEIF
2012)

May 4th, 2012


Authors

Marcelo M. Pinheiro
14
, Igor L. E. Macêdo
1
, Igor L. O. Souza
1
,
Thiago S. Hohlenweger
13
, Paulo R. R. Leite
1
, Adriano L.
Spínola
1
, Herbert Monteiro
1
, Raphael A. Dourado
2
,
Leobino

N.
Sampaio
4
, José A.
Suruagy

Monteiro
2

and

Joberto

S. B.
Martins
1



1
Salvador University (UNIFACS)


2
Federal University of
Pernambuco

(UFPE)

3
Federal Institute of Bahia (IFBA)


4
Federal University of Bahia (UFBA)


FIBRE
-
BR
will possibly use
three different control and
monitoring frameworks in its nine
islands


OFELIA
Control
Framework


cOntrol

and Management Framework (OMF)
and


ProtoGENI


Each one takes a different approach in addressing I&M
requirements and demands


Each CMF has its own monitoring capabilities


How to put all these together?

M
otivation


An Instrumentation and Measurement Architecture
Supporting Multiple Control Monitoring Frameworks


Our
target
is:


to
provide, possibly, with a maximum reuse of the
available CMFs
I&M services over
a new integrated
and federated network
structure;


To provide instrumentation and monitoring
considering different I&M Services through FIBRE
-
BR (Monitoring Orchestration);


Multiple
CMFs I&M data
integration.

Goal

FIBRE
-
BR I&M
ARCHITECTURE

FIBRE
-
BR
I
&
M Persistent
Data Repository
OFELIA
Monitoring
Data
MDIP
Measurement
Data
Integration
Point
Aggr
03
Aggr
02
Aggr
01
OFELIA CMF
OFELIA
Control
Commands
ProtoGeni
Monitoring
Data
MDIP
Measurement
Data
Integration
Point
Aggr
03
Aggr
02
Aggr
01
ProtoGENI CMF
ProtoGeni Monitoring
Facilities
ProtoGeni
Control
Commands
OMF
Monitoring
Data
MDIP
Measurement
Data
Integration
Point
Aggr
03
Aggr
02
Aggr
01
OMF CMF
OMF Monitoring
Facilities
OMF
Control
Commands
Experimenter
Researcher
Network
Manager
Visualization
/
Portal Service
Security Services
Orchestration
&
Configuration Service
FIBRE
-
BR
Policy
Repository
OFELIA Monitoring
Facilities
FIBRE
-
BR
I
&
M
SERVICES
FIBRE
-
BR I
&
M
SERVICES
FIBRE
-
BR I
&
M
SERVICES
FIBRE
-
BR I
&
M
SERVICES
1

2

3

5

4

Measurement

Data

Integration

Point

(MDIP
)

-
conforms

the

collected

data

from

the

available

CMFs

to

FIBRE
-
BR

I&M

standard

format

(NM
-
WG),

representation

and

distribution

(including

visualization)
.


-
This

service

includes

all

measurement

data

processing

related

aspects

such

as,

message

format,

message

transport

protocol

and/or

service,

access

privileges

and

common

data

storage

or

on
-
the
-
fly

data

distribution
.

1

The

security

and

police

service

will

use

the

global

definitions

implemented

and

controlled

by

the

ClearingHouse

(CH

definition

-

is

both

an

entity

and

a

system

consisting

of

software,

operations,

and

policy

to

broker

trust

between

federation

partners
.
)

Some

services

that

are

going

to

be

used

by

the

I&M

Architecture
:


Trust

relationship

(CA,

SASL,

etc
)


Identity

credentials


Integrated

authentication/authorization


Federation

level

policies


Slice

behavior


Data

access

policy


Policy

enforcement


FIBRE
-
BR

policy

document


2

The

Orchestration

and

Configuration

Services

act

on

behalf

of

the

users

allowing

them

to

configure,

to

define

measurement

points,

and

to

orchestrate

these

measurement

data

collection

facilities

according

to

each

individual

CMF
.


Implementation

initial

ideas
:

-
Use

XML

pub/sub

messaging

service,

based

on

XMPP

server

-
Currently

supported

by

OMF

-
IMF

@

GENI

3

The

I&M

Portal

main

functionality

is

to

provide

a

user

friendly

interface

to

control

and

access

the

measured

data,

according

to

a

defined

policy
.

Data

visualization
:

-
From

real
-
time

experiments

-
From

data

stored

Persistent

Data

repository

in

each

individual

CMF

(I&M

perspective)

-
Verify

privileges

access

-
Available

only

to

authorized

users

4

The

architecture

has

a

storage

strategy

that

allows

users

to

retrieve

data

from

their

own

or

from

others

previous

experiments,

according

to

their

access

privileges
.

The

persistent

storage

option

is

an

experimenter

decision

that

must

comply

with

FIBRE
-
BR

retention

policy
.

Implementation

steps
:



MDIP

will

be

in

charge

of

saving

it

persistently


Data

retention

policy


MySQL/RRD/SQLLite
3
/
PostgreSQL

database

(access

from

I&M

solution)


I&M

Standard

storage


Each

CMF

will

keep

its

storage

mechanism
.

I&M

will,

eventually,

store

it

centrally

or

access

it

based

on

users

demand

and/or

privilege


Logs

storage

5

OML Overview
and

OML
-
MDIP
Proposal

13


OML
was

first

developed

as
the

measurement

component

of

OMF


Today

is a stand
-
alone

project

(
independent
)


Shortly
, it is a framework (set
of

libraries
) to
collect

and

store

measurements


OML
Introduction

14


OML MDIP
consists

basically

of

one

service
: OML MA
(OML
Measurement

Archive
)


It’s

responsible

for
receiving

and

sending

the

requests

and

responses


When

it
receives

the

requests
,
the

MA queries OML Server DB
and

then

uses standard
perfSONAR

messages

to

communicate

with

FIBRE
-
BR I&M Services


The response is
sent

to
the

MDIP, in
order

to

be

stored

persistently

in FIBRE
-
BR I&M
repository

(
if

demanded
)

OML MDIP
Proposal
(1)

15

OML MDIP

Proposal
(2)

16

-
FIBRE
-
BR
I&M Architecture is designed to configure and
collected data from different CMFs.

-
It is intended to be an instrumentation and measurement
evolutionary architecture

-
There are a lot of challenges to overcome!

Conclusions

Thank

you

/ Obrigado

Backup slides

ProtoGENI

I&M
Architecture

and

Tools


There are several GENI I&M related projects compatible
with
ProtoGENI
. The most important ones are:


INSTOOLS


LAMP


OneTimeMeasure


S3Monitor


Some of them are complimentary and will later be
integrated to compose a GENI Integrated I&M
Framework

Introduction

Instrumentation Tools

(INSTOOLS)


INSTOOLS’ high
-
level

goal
:


Make
it easy for users to
see what
is
going on in their
experiment



i.e., make it trivial to
monitor a
slice


What can INSTOOLS

measure?


Note that INSTOOLS is

concerned only about

passive measurements


INSTOOLS’ philosophy


Don’t reinvent the wheel



Overview


Architecture

(overview)

Experimenter

Measurement

Plane

connections

Data Plane

connections

MC Portal: single interface for
all

MCs

in a
slice

Slice’s

nodes

(
instrumentized

to

act

also

as MPs)

Measurement

Controller

(MC) = GENI MAP+MC

Automatically

deployed

by

INSTOOLS (
at

least

one

per
aggregate
)


INSTOOLS Web Portal

LAMP


LAMP stands for “
Leveraging and Abstracting Measurements
with
perfSONAR



The main goal is to “create
an instrumentation
and
measurement
system, based on
perfSONAR
, for use by
experimenters
on
ProtoGENI



Which

tools does LAMP
support
?


OWAMP,
BWCTL,
Ganglia
,

PingER
, NTP;
ps
-
BUOY MA;
etc


So
...
what’s

the

difference

between

LAMP
and

pS
-
PSToolkit
?


LAMP adapted
perfSONAR
-
PS software suite to recognize
GENI’s
Authentication and Authorization
model and
infrastructure


Added Ganglia
as a host monitoring
solution


Added
distributed configuration through annotations in the topology
stored
in
UNIS


These

annotations

make
it easy for users to save their slice
configuration and load
it at
a different
slice

Overview

Architecture

(a
broader

view
)

ProtoGENI

UNIS

Slice

1

Slice

2

Slice

n

Node with LAMP tools

Node with LAMP tools

+ LAMP Portal enabled

UNIS
keeps

information

about

ALL

slices

Note that it is possible to
have more than one node
running the Web Portal

Not

every

node
has

to

be


instrumentized


This

node
has

only


Measurement

Plane
interfaces”


LAMP Portal

Each

node
has

as
set
of

(
possible

different
)
measurement

tools
enabled

OneTimeMeasure

and

S3Monitor


OnTimeMeasure


Is “an
on
-
demand measurement service used in
forecasting,
anomaly detection, and fault
-
location

diagnosis in GENI
experiments and GENI operations
.”


Can

be

integrated

with

INSTOOLS


S3Monitor


Has a flexible
design
that allows
easy “plug in” of new network
measurement tools


Based

on

𝑆
3

(
Scalable Sensing Service for
PlanetLab
)


Focused on measurement of large networked systems


Already integrated with INSTOOLS

OneTimeMeasure

and

S3Monitor

Integrating with

FIBRE
-
BR I&M architecture

INSTOOLS MDIP

ProtoGENI

Fibre

I&M
Architecture

Slice

(INSTOOLS)

INSTOOLS

Meas

Controller

RRD

ProtoGENI

MDIP

RRD

Collector

FIBRE I&M Data
Repository

1

2

SQL DB

SQL

Collector

1.
MC collects measurement data from the MPs

2.
MDIP (through his collectors) collects measurement data, makes any
necessary format adjustments and

3.
Stores the data in the permanent repository (if demanded)

4.
Measurement data can be accessed through the Visualization Portal

3

Visualization

Portal

4

LAMP MDIP

ProtoGENI

Fibre

I&M
Architecture

Slice

ProtoGENI

MDIP

FIBRE I&M Data

Repository

NMWG

Generic

Collector

UNIS

MP

MP

LAMP Web Portal

SNMP MA

ps
-
BUOY MA

Ganglia

MA

PingER

MA

MA
-
specific


defs

1.
Fetch experiment description from UNIS

2.
Start copying measurement data from MAs

3.
Stores the
data
in the permanent
repository (if demanded)

1

2

3

Visualization

Portal

OFELIA I&M
Architecture


Igor Luiz (UNIFACS)

35

NOWADAYS



Backend Infrastructure Monitoring (servers, switches, links)


-

ZENOSS (not integrated to OCF)



OFELIA
´
s experiment monitoring:


-

Still under discussion and development

OFELIA
I&M STATUS

36


FIBRE I&M
possible

alternatives

for OFELIA CMF are (in
discussion
):


Focus
on

FlowVisor

and OF
-
related

basic

measurements

parameters

(
pragmatic

approach)


Incorporate

current

OFELIA
monitoring

developments

(
need

to

identify

them
)


Align

with

OFELIA
current

developments


OFELIA & FIBRE I&M



OFELIA I&M
technical

possibilities
/
discussions
:


OML


sFlow


perfSONAR


LAMP


... !?