Raptor Technical Details

streakconvertingSoftware and s/w Development

Dec 13, 2013 (3 years and 8 months ago)

74 views

Raptor Technical Details

Outline


Workshop structured by Raptor workflow


Raptor Event model.


ICA log file parsing


ICA/MUA event storage


ICA event release


MUA batch event persistence


MUA statistical processors


WEB overview


WEB and MUA hookup and operation


Live walkthrough of creating a new event time

Raptor Event Model


Raptor uses a concrete Java (POJO) event
model e.g. each event is a ‘real’ class (with
appropriate hierarchy).

e
ve
n
t
T
i
me
....
Ev
e
n
t
p
ri
n
ci
p
a
l
N
a
me
....
A
u
th
e
n
ti
c
a
ti
o
n
Ev
e
n
t
binding
....
Sh
i
b
b
o
l
e
th
I
d
p
A
u
th
e
n
ti
c
a
ti
o
n
Ev
e
n
t
p
ro
t
o
co
l
....
Sh
i
b
b
o
l
e
th
Sp
A
u
th
e
n
ti
c
a
ti
o
n
Ev
e
n
t
se
ssi
o
n
I
d
....
Ezp
r
o
x
y
A
u
th
e
n
ti
c
a
ti
o
n
Ev
e
n
t
e
ve
n
t
T
i
me
....
A
g
g
r
e
g
a
te
d
Ev
e
n
t
Raptor Event Model

Raptor Event Model


To create a new event class, you *must* currently do
the following.

1
Check
-
out the current raptor
-
information model from
svn
.

2
Create a new class in the event package

3
Add necessary fields

4
Create a
toString

method (for debugging)

5
Create a hash and equals method (so duplicates can be
understood


business key).

6
Configure hibernate xml file for new type.

7
Use maven to recompile the library, rename to same as
existing raptor
-
information
-
model and replace in lib dir
of ICA, MUA and WEB.

Log File Parsing


Event
-
parse specifies a number of
BaseEventParsers
. Each currently a log file
parser.


Each parser must determine from line filters
which lines are of interest.


Each parser has associated format, which
gives mappings from values in the log file to
the internal model field name.



Log File Parsing Flow

Event Storage


In memory on ICA and DB backed in MUA.
Both share similar flow…but.


ICA uses a slightly different storage semantic.
Incremental checks for efficiency, whereas
MUA does not make assumptions about
temporal ordering, just checks for existence
and if not found stores.

Incremental Event Storage

Persistent Event Storage

ICA Event Release


Releasing events to configured
MUAs

(more
than one, although small technical bug that
needs fixing)


Each MUA is an endpoint and accepts a particular
type of
event(s
).


Different release policies to different
MUAs


Number of events


Elapsed time


Or combination


ICA Event Release


Events release is called
every 30
seconds.


High level locks exist to stop release from running if
capture (or any other operation) is running.


Applicable events are sent to each endpoint
individually.


Events filtered by time


Events filtered by type


Events filtered by resource category (internal/external)


Events sent over as SOAP messages over HTTPS using
apache CXF


Events class marshaled in and out of XML using Aegis
binding.

ICA Event Release

MUA Batch Event Parsing


ICA uses an in
-
memory Set to hold events. If ICA
dies, so do the events it holds. But restarting the
ICA reparse and re
-
send all events. MUA will then
discard on the hash id of the event.


MUA uses a persistence layer to map Events to
tables in the db.


Each event requires a Hibernate ORM configured.


Hibernate then maps from the object through to the
correct SQL dialect on the configured database.

MUA Batch Event Parsing


Events are received from the ICA through its
exposed SOAP endpoint.


Events are stored asynchronously via a
background worker thread.


Each event is passed through the attribute
expansion engine
before

storing.


Events are stored through a similar event
handler as the ICA


only configured to talk to
the DAO layer.



MUA Batch Event Parsing

I
C
A

se
n
d
s
e
ve
n
t

se
t
C
re
a
t
e

a

n
e
w

t
ra
n
sa
ct
i
o
n

a
n
d

b
a
ckg
ro
u
n
d

w
o
rke
r
t
h
re
a
d

f
o
r
sa
ve
F
o
r
e
a
ch

e
ve
n
t
,

se
n
d

i
t

t
h
ro
u
g
h

t
h
e

a
t
t
ri
b
u
t
e

a
sso
ci
a
t
i
o
n

engine
Pa
ss
e
ve
n
t

t
o

st
o
ra
g
e

h
a
n
d
l
e
r
(se
e

p
re
vi
o
u
s)
MUA Statistical Processors


MUA stores all the statistical units and
processors necessary for computing stats. This
does not reside within the Web component.
They are like templates.


Statistical
-
units
-
custom stores user
configurable (through the web UI) stats.


Statistical
-
unit
-
system stores the stats the web
interface calls upon to do the dashboard
statistics.

MUA Statistical Processors


New stats can be configured in the XML, and
the MUA needs to be restarted for them to
take affect.


MUA compiles a capabilities object with those
statistics loaded and passes this to the Web
component so it knows what is available.


Each statistic has a number of properties for
series definition, display and post processors.

MUA Statistical Processors


Post processors can be attached to a statistic
to perform certain functions e.g. add SAML
org name, or remove rows from result.


Each processor is defined in the statistical
-
processors.xml

file.


Processors are instantiated and added to a
statistic whenever the stat is created or
updated.


W
e
b

a
sks
MU
A

f
o
r
st
a
t
i
st
i
ca
l

ca
p
a
b
i
l
i
t
i
e
s
We
b
Login
T
o
We
b
MU
A
getCapabilities
I
n
vo
ke

o
r
u
p
d
a
t
e
/
i
n
vo
l
e

st
a
t
i
st
i
c
u
p
d
a
t
e

a
n
d

i
n
vo
ke

st
a
t
i
st
i
c
u
p
d
a
t
e

a
n
d

i
n
vo
ke

st
a
t
i
st
i
c
V
i
e
w

a
l
l

a
va
i
l
a
b
l
e

st
a
t
i
st
i
cs
u
p
d
a
t
e

st
a
t
i
st
i
ca
l

unit info e.g.
p
e
ri
o
d
,

n
a
me
,

se
ri
e
s
cl
e
a
r
a
n
d

re
-
i
n
st
a
n
t
i
a
t
e

st
a
t
i
st
i
ca
l

p
ro
ce
sso
rs
fin
d

st
a
t
i
st
i
ca
l

u
n
i
t

to update
fin
d

st
a
t
i
st
i
c
b
y
n
a
me
f
o
r
e
a
ch

se
ri
e
s
d
o
cre
a
t
e

b
u
cke
t
s
b
e
t
w
e
e
n

st
a
rt

a
n
d

e
n
d

t
i
me
q
u
e
ry
d
a
t
a
b
a
se

f
o
r
e
ve
n
t
s
o
f

g
i
ve
n

t
yp
e

b
e
t
w
e
e
n

g
i
ve
n

dates
f
o
r
e
a
ch

b
u
cke
t
a
d
d

re
su
l
t
s
t
o

b
u
cke
t
st
o
re

se
ri
e
s
re
su
l
t
s
Se
n
d

t
o

p
o
st

p
ro
ce
sso
rs
f
o
r
e
a
ch

se
ri
e
s
d
o
se
n
d

se
ri
e
s
d
a
t
a

t
o

p
o
st
p
ro
ce
sso
r
C
o
n
st
ru
ct

a

R
a
p
t
o
r
G
ra
p
h

Mo
d
e
l

f
ro
m
st
a
t
i
st
i
ca
l

re
su
l
t
s
co
n
ve
rt

ra
p
t
o
r
g
ra
p
h

mo
d
e
l

t
o

png
MUA Statistical Processors



Most heavy lifting on the database layer e.g.
groupby

and temporal queries


However, results are loaded into the MUAs memory to:


Allow post processors to act on them


Construct a graph model so that the web interface can
construct a graph image from them.


This is one of the primary reasons the MUA eats
memory (not really on the event storage lifecycle).


The Web interface also takes some time and memory
to construct a graph image


using
JFreeChart
.

Web Overview
-

Technologies



The web interface is a Java servlet, running spring
web flow, Java Server Pages, and Java Server
Faces for UI components.


Also used
Richfaces

components


Follows Model (Java
Pojo
), View (JSP/JSF),
Controller (spring
webflow
) paradigm.


In reality, view components get decoded to HTML
and
Javacript

(lots of) which makes calls and (Ajax
or full page refresh) to the Java Servlet (which
runs on Jetty).



Web Overview
-

Functions


Can attach to any MUA (providing keys are in
place)


Runs set of dashboard statistics


Can perform batch uploads to the MUA (limited
file size due to non
-
streaming transport).


Can invoke any of the
retrived

MUA statistics


Converts the graph model sent back by the MUA
into a graph and table


Can convert the table and or graph into a PDF,
CSV or Excel file format.

Web Overview


Interaction with MUA


When attaching to an MUA (when it starts, or
the user requests it), Web asks the MUA for its
capabilities. These are:


Statistical units


Statistical processors


Event information e.g. no. of events per type.


Suggestion values


Resource Information.

Web Overview


Interaction with MUA


Capabilities information is used throughout
the UI e.g.


In the setup page to show resource classifications
and event information


In the graphs page to list possible statistics and
their values


Suggestion values are used in the drop downs for filters


The dashboard stats creations (explained next)


Web Overview


Interaction with MUA


The definition of what is run by the dashboard page is
stored in the web configuration (dashboard
-
statistics.xml
).


This stores a list of what type of statistics to run, over what
time period and for what event type.


The actually statistic it calls in defined on the MUA in the
statistical
-
units
-
custom.xml

file.


So its almost a meta
-
template for a statistic.


These can be amended or turned on and off there.


Some stats can take a long time to run e.g. unique users.

Web Overview


Interaction with MUA


Graphs page shows those statistical units
retrieved from the attached MUA.


Clicking on the statistic sends an invocation
request to the MUA using the statistic name as
the stat to invoke.


Expanding the statistical parameters allows you
to alter the statistic. By clicking on update it
sends the entire set of statistical parameters back
to the MUA so the MUA can update it. It then
also sends an invoke request to perform the
statistic.

Web Overview


Batch Upload



Web loads each specified file into an in
-
memory object.


This object is then send (via CXF, SOAP) to the
MUA.


The file can not be streamed due to the SOAP
binding raptor uses.


Hence a limit on the number of files.