FIRST-D7.1 Integration Infrastructure Release v1.0

Project Acronym:

FIRST

Project Title:

Large scale information extraction and
integration infrastructure for supporting
financial decision making

Project Number:

257928

Instrument:

STREP

Thematic Priority:

ICT
-
2009
-
4.3 Information and Communication
Technology


D
7.1

Integration
Infrastructure Release

Work Package:

WP
7

–

Integrated Financial Market Information
System

Due Date:

31
/
03
/
2012

Submission Date:

3
0/03/2012

Start Date of
Project:

01/10/2010

Duration of Project:

36 Months

Organisation Responsible
for

Deliverable:

ATOS

Version:

1.0

Status:

Final

Author
(s)
:

Mateusz Radzimski

Murat Kalender


ATOS

ATOS


Reviewer(s):

Irina Alic

Julia Zillies

UGOE

UGOE

Nature:


R
–

Report

P
–

Prototype


D
–

Demonstrator

O
-

Other

Dissemination level:


PU
-

Public



CO
-

Confidential, only for members of the
consortium (including the Commission)


RE
-

Restricted to a group specified by the
consortium (including the Commission Services)

Project co
-
funded by the European Commission within the Seventh Framework Programme (2007
-
2013)






D7.1



Revision history

Version

Date

Modified by

Comments

0.1

20
/
0
2
/
201
2

Mateusz Radzimski
(ATOS)

First

version of ToC
provided

0.2

20/02/2012

Mateusz Radzimski,
Murat Kalender
(ATOS)

Description of

the

integration component

0.3

2/03/2012

Mateusz Radzimski

Update of “Implementation
details” section.

0.4

5/03/2012

Mateusz Radzimski

Executive summary and
Conclusions
, Annex 1

1.0

30/03/2012

Mateusz Radzimski

Addressing reviewers’
comments
. Final formatting.







D7.1


Copyright © 201
2
,
FIRST Consortium

The FIRST Consortium (
www.project
-
first.eu
) grants third parties the right to use and distribute
all or parts of this document, provided that the FIRST project and the document are properly
referenced.

THIS DOCUMENT IS PROVIDED BY THE COPY
RIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTOR
S BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY T
HEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
DOCUMENT, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

----------------





D7.1


Executive Summary

Based on the
d
esign and simulation studies c
onducted throughout
WP2
, implementations of the
integration infrastructure prototypes have been started. This deliverable presents technical
details of the lightweight messaging middleware component used for FIRST pipeline int
egration.
It is in line with the conceptual and technical design and the scaling strategy (see
(FIRST D2.2
Conceptual and technical integrated architecture design, 2011)

and
(FIRST D2.3 Scaling
Strategy, 201
1)
).

This deliverable is accompanied by the software prototype that
realizes
stream
-
based
integration
. The main objective is to provide
a
necessary technical description for configuring
and running the software component.

This software is also being
used by the early integrated version of FIRST components

(FIRST
D7.2 Early prototype of Integrated Financial Market Information system, 2012)

and may likely
be
subject to minor improvements

within the development and integration

of the overall FIRST
inf
rastructure.

O
ther aspects of FIRST integration are subject of the de
liverable D7.2 and future
WP7 deliverables.







D7.1



Table of Contents


Executive Summary

................................
................................
................................
......

4

Abbreviations and acronyms

................................
................................
.......................

7

1.

Introduction

................................
................................
................................
............

8

1.1.

Purpose of this document

................................
................................
..................

8

2.

Lightweight integration component
................................
................................
......

9

2.1.

Implementation details

................................
................................
......................

9

2.2.

C# Integration

................................
................................
................................
..

11

2.2.1

Instal
lation

................................
................................
................................

11

2.2.2

Configuration

................................
................................
............................

13

2.2.3

API for C#
................................
................................
................................
.

14

2.2.4

Sample code

................................
................................
............................

15

2.3.

Java Inte
gration

................................
................................
...............................

15

2.3.1

Installation

................................
................................
................................

15

2.3.2

Configuration

................................
................................
............................

17

2.3.3

API for Java
................................
................................
..............................

18

2.3.4

Sample code

................................
................................
............................

18

3.

Hardware resources

................................
................................
.............................

20

4.

Conclusion

................................
................................
................................
............

21

References

................................
................................
................................
...................

22

Annex 1.

Example configuration file for C# component

................................
.....

23

Annex 2.

Example configuration file for Java component

................................
.

24

Annex a.

Main configuration

................................
................................
....................

24

Anne
x b.

Logging configuration

................................
................................
...............

24

Annex 3.

Configuration of the messaging test scenario

................................
....

26

Annex a.

Configuration of components mimicking WP3 behaviour (data emitters)

.

26

Annex b.

Configuration for mimicking WP4 behaviour (data receivers and emitters)

26

Annex c.

Configuration for mimicking WP6 behaviour (data receivers only)

...........

28


Index of Figures

Figure 1:
Technical details of integration component

................................
................................
.....

9

Figure 2: Sample load balancing setup (2x and 1x mean component’s throughput as a numb
er of
messages processed at once).

................................
................................
................................
........

10

Figure 3: Setting up ZeroMQ libraries for C# implementation

................................
....................

12

Figure 4: Adding ZeroMQ references to the C# project in Visual Studio

................................
....

12

Figure 5: Adding ZeroMQ libraries to the C# project in Visual Studio (2)

................................
..

13

Figure 6: Setting up ZeroMQ libraries for Java implementation

................................
..................

16

Figure 7: Adding ZeroMQ libraries to the Java project in Eclipse

................................
...............

17






D7.1


Index of
Listings

Listing 1: Example of C#

API

................................
................................
................................
.......

15

Listing 2: Example of Java API

................................
................................
................................
....

19








D7.1


©
FIRST

consortium


Page
7

of
28

Abbreviations and acronyms


IFMIS

Integrated Financial Market Information System

WP

Workpackage

ID

Identifier

DB

Datab
ase

DLL

Dynamic Link Library

API

Application Programming

Interface

JVM

Java Virtual Machine






D7.1


©
FIRST

consortium


Page
8

of
28

1
.

Introduction

Disclaimer: for reader’s convenience some fragments of this deliverable are repeated from WP2
deliverables.

1
.
1
.

Purpose of this document


This document presents
implementation and configuration details

of the integration
c
omponent

developed withi
n WP7.

The messaging component is implemented according to the design
guidelines from
(FIRST D2.2 Conceptual and technical integrated architecture design, 2011)

and is supporting scaling techniques as described in
(FIRST D2.3 Scaling Strategy, 2011)
.

We are also presenting a testing environment that mimics the pipeline configuration, for
messaging testing purposes.

Furthermore, t
his document is
also
accompanied by the
prototype
implementation source code
.


Please note that
t
he goal of this document is to present an isolated pipeline integration
approach. For a view of the current integrated set of components (where messaging
components plays an important role in the integration), see
(FIRST D7.2 Early prototype of
Integrated Financial Market Information system, 2012)
.

In the further
chapter

we
introduce the integration component and provide

technical details
of
implementation

and setup.

It will also cover
API

binding for both Java and C#. Next,
we
describe short overview of the
already running
deployment platform
.

The last chapter presents
conclusions from development of the integration component.
All configuration examples have
been put in annexes and a
re ref
erred from within the deliverable
.






D7.1


©
FIRST

consortium


Page
9

of
28

2
.

Lightweight integration

component

This chapter explains the implementation of the integration component
that provides
facilities for
pipeline components integration.

As the result of
the simulation studies accomplished within
WP2 and described in
(FIRST D2.2 Conceptual and technical integrated architecture design,
2011)
, we follow the messaging approach

as a way to process data streams through pipeline
components
.

The messaging approach adopted in FIRST can be simply explained as
data (message)

exchanging between components. T
echnically
it
means that

components in the FIRST pipeline
are
passing

messages

(i.e. web documents)

between each other
. They are
streamlined

so

that
dataf
low throughput c
an be optimal and no single component needs to orchestrate the whole
process. Rather each component works
individually by processing received messages and
forwardin
g results to the next component in line.

Th
e following subsections present

implementation and configuration details.

2
.
1
.

Implementation details

The messaging component has been implemented
based on

the

ZeroMQ
1

library.

The
design

and
performanc
e

considerations

have

been

carried on

in
(FIRST D2.2 Conceptual and technical
integrated architecture design, 2011)
.

This chapter describes
the
current implementation
features

of this component
from

a

practical point of view, as it is used
to
interconnect

p
ipeline
components.

Since pipeline components are developed by different partners using different technologies (
e.g.
WP3 components using .N
ET platform and WP4 using Java)

the messaging component needs
proper API
binding
s

to support those lan
guages
.

Therefore
the
component has

been developed
in two versions: C# and Java, having
the same functio
nalities

and corresponding API

for each
language.

The single component can be configured to be either emitter or receiver

of messages

or both
.

The
ZeroMQ library
is system
-
dependent and therefore it must be compiled for different
operating system
s
. In
the current

case it
has been compiled for both 32
-
bit and 64
-
bit platforms
.



Figure
1
: Technical details of integration comp
onent
2

The most important feature
i
s

the
fast exchange of messages between two (or more)
messaging components.

It currently supports

both

messaging scenarios

that can be configured
using

configuration file (
for details
see
paragraphs
2.2.2

for C# and
2.3.2

for Java
)
:




1

ZeroMQ library: http://www.zeromq.org/

2

As designed in
(FIRST D2.2 Conceptual and technical integrated architecture design, 2011)




D7.1


©
FIRST

consortium


Page
10

of
28



Request/Reply
:

a
classical

scenario

where
sender and receiver exchange messages
one by one in a blocking fash
ion

where

each message is confirmed before the next one
is sent.



Pipel
ine
: messages are pushed in a stream
-
based manner from
the
emitter to the
receiver.
Messages are sent asynchronously,
without
an
explicit request from the
message consumer
.
On the receiver part, messages are kept in a queue

before they are
consumed.

This a
pproach performs much faster than Request/Reply, but at a cost of
additional

synchronisation

and
this approach
pose
s

a
risks of queue overflow
(and
message dropping)
when too big message peak
s

occurs
.

In a simple scenario, overflow
is handled by using
an
additional control channel

between
the
message emitter and
the
message receiver
to notify
the queue state
.

In addition

to that,
the
mess
a
ging component
support
s

multiple senders and receivers and in
such case
s

load balancing between the nodes is performed

according to

FIRST D2.3
deliverable
(FIRST D2.3 Scaling Strategy, 2011)

as shown in a simplified version with two
receivers on
Figure
1
.

L
oad balancing aims at directing messages to the least occupied
component, thus distributing overall processing load as evenly as possible.

More
technical
details have been exposed in D2.3

deliverable
.


Figure
2
:
Sample load
balancing setup

(
2
x and 1x mean

component’s throughput as a
n
umber
of messages

process
ed

at once)
.


Figure
2

presents a more complex test scenario setup
of
the
compon
ents
for load balancing
experiments.
Messages are produced within two WP3 components, and are sent to four WP4
consumers. At this moment
a
load balancing algorithm performs a distribution of messages
between all receiving components.
When there are multipl
e consumers and producers in the
pipeline, we need a scheduling algorithm. We assume that message consumers are identical
and consume
a
similar amount of messages at a time.
If

the queue of a message consumer is
full, an “overflow” message is sent through
the control channel.
All “overflow” messages are
sent to each message producer therefore every producer knows how many consumers have
their queue full.
The flow of the messages is controlled
by
using IDs and consumer/producer
lists.
In
the

test scenario

(
two

WP3
producer
s

and
four

WP4
consumer
s)

if one

consumer sends
an
overflow message all producers
will
continue messaging.
If the
second

overflow message is
sent,
the
producer with ID value 1
will

stop sending messages.

After WP4 finishes

processing
messag
es
, results are sent up the pipeline to
the
WP6 components. In this scenario
WP6
components

perform as a “sink” receiving messages from multiple producers. Configuration
details of this sample scenario are presented in
Annex 1
.

Please note that
the
presented
scenario is for messaging middleware
experimenting

testbed

only

and is not the exact target
FIRST deployment scenario
.




D7.1


©
FIRST

consortium


Page
11

of
28

Other implemented features of the messagin
g component:



Handling failures:
T
o prevent data loss the messaging component is able to redirect all
overflowing messages in queues to
a
persistent storage
. In the target FIRST deployment
scenario this feature will not be used, as the pipeline will be dime
nsioned according to
the traffic to be processed.




Multithreading
:
The
messaging library has been multithreaded for performance reasons.
We use separate threads for sending
and receiving messages

and for control channel
messaging
.



Queue data structure
:
O
n
the sender and receiver, all messages are store
d

in a
FIFO
queue
s
.



Logging:
The
component provides logging through the external logging library (Log4j for
Java and log4net for C# implementation)

2
.
2
.

C
# Integration

The messaging component provides

an

implement
ation endpoint (emitter
and

receiver) for use
with

a

C# platform.

The f
ollowing
sub
sections explain
the
configuration and installation details of
the C# implementation.

2
.
2
.
1

Installation

The Messaging library composes of one main
DLL

file and ha
s
the following
three
dependencies
:

Z
ero
MQ
, log4net and
ZeroMQ

binding for C# language. You can get the latest
versions of these libraries from
svn://first.ijs.si/ATOS/32/library/C#
1

for 32bit
DLLs
and
svn://first.ijs.si/ATOS/64

for 64bit
DLLs
.

For the sake of this deliverable, a current
snapshot of the code and libraries can be obtained under:
http://project
-
first.eu/content/d71
-
integration
-
infrastructure
-
release
.

List of compiled artifacts:



Messenger.dll (Messaging library)



log4net.dll (Logging

library for C#)



libzmq.dll (
ZeroMQ
library)



clrzmq.dll (
ZeroMQ
library for C#)













1

All the
SVN
-
referred files and code can be also found in ZIP file attached

to this deliverable.




D7.1


©
FIRST

consortium


Page
12

of
28

To integrate these libraries in Visual Studio:

1
.

Add libzmq.dll to the System path or to main programs’ directory (Executable).


Figure
3
: Setting up ZeroMQ libraries for C# implementation


2
.

Right click on the References in Solution Explorer window and select “Add Reference”


Figure
4
: Adding ZeroMQ
references

to the C# project

in Visual Studio







D7.1


©
FIRST

consortium


Page
13

of
28

3
.

Click the Browse t
ab and select the checked out 3
DLL

files


Figure
5
: Adding ZeroMQ libraries to the C# project
in Visual Studio
(2)

2
.
2
.
2

Configuration

The Messaging library gets parameters via configuration files. The following parameters must
be added to
the
app.config file in order to use
the
messaging library:



ID: Id of the program to distinguish from other components in the pipeline
.



ReceiverNumber:

Number of components

that

send messages to the component
.



ProducerNumber: Number of components

that
produce messages in parallel
.



MessageSendAddress:

Port to send messages
.



MessageReceiveAddress: Port to receive messages
.



SendLoadBalancingAddress: Port to

send load balancing commands to
the
message
producer to stop receiving messages
.



ReceiveLoadBalancingAdress: Port to receive load balancing commands from

the

message consumer to stop sending messages.



FinishPublish: Port to publish
the
finish command to t
erminate messaging
.



FinishReceive: Port to listen
to the
finish command
.



MessagingType:
S
upport for various messaging types:

o

0=Pipeline, 1=Request and Reply
.



MAX_QUEUE_SIZE: In pipeline messaging type,
the
message sender doesn’t wait
for
the
message receiver to consume these messages.
The m
essage sender stops sending
messages, if
the
queue size exceeds the max value.




D7.1


©
FIRST

consortium


Page
14

of
28



MIN_QUEUE_SIZE: Minimum queue size to send continue message
s

to
the
message
producer
.



IGNORE_QUEUE_OVERFLOW: Ignores messages wh
en
a
data peak occurs.



Broker:
If the
receiver can’t consume all messages, messages can be sent to a message
b
roker to consume them later on
:

o

0=File storage, 1=Active
MQ
1



ACTIVEMQ: Active
MQ

messaging port
.



QueueName: Name of the queue used between WP3 and W
P4
.



InFileStorageAddress: When messaging is terminated before

all messages are
consumed
,

the

messages are stored in files in this directory to consume later on
.



OutFileStorageAddress:
If the
message send queue exceeds the maximum value or
messaging termina
tes before sending all messages to the consumer, messages are
stored in files in this directory.



maxFileStorageNum: Maximum number of messages that can be stored in files.



DBLoggingReceiver: Messaging port for logging messaging in a database
.



DB_LOGGING: E
nables logging messages in a database
.



log4net: Contains the logging configuration settings. You may change the level value
between “info and debug” to change logging details. Current configuration settings write
logging information both in console and in
a file (messenger.log).

You can use

the app.config file located under following
locations:

svn://first.ijs.si/ATOS/64
/Messenger C#/Test
/app.config

svn://first.ijs.si/ATOS/64
/Wp3 Test/Test.exe.config

svn://first.ijs.si/AT
OS/64/Wp6 Test/Test.exe.config

The sample configuration file used by
the
Messanger component

is presented in
Annex 1
.

2
.
2
.
3

API

for C#

Currently Messaging API contains one class Messenger. This class
has three methods open for
the messaging purposes:



void sendMessage(String message)
: Sends
an
input message to the message
consumer



String getMessage()
: Returns one message file from

the

received messages queue.



void stopMessaging()
: Sends
a
finish message

to the message consumer and
terminates messaging.



bool isMessagingFinished()
: Returns true if
the
finish command is received from one of
the components in the pipeline.




1

ActiveMQ
broker
-
based messaging system:
http://activemq.apache.org/




D7.1


©
FIRST

consortium


Page
15

of
28

2
.
2
.
4

Sample code

There is
a
test project, which sends test messages to the message
listener. You may use this
project for testing purposes (
svn://first.
ijs.si/ATOS/Messenger C#/Test
).

The following code block sends received messages

to message listener components:


Messenger

messenger =
new

Messenger
();

while

(!
messenger.isMessagingFinished())

{


String

inMessage = messenger.getMessage();



if

(inMessage !=
null
)


{



logger.Info(
"Gets message number: "

+ inMessageNum++);



messenger.sendMessage(inMessage);


logger.Info(
"Sends m
essage number: "

+ inMessageNum);


Thread
.Sleep(delay);


}

}

Listing
1
: Example of C# API

2
.
3
.

Java Integration

The second API implementation is the Java binding. The messaging component provides an
implementation endpoint
(emitter and receiver) for use with a JVM

(Java)
. The following
subsections explain the configuration and installation details of the
Java

implementation

and it is
analogical to the C# version.

2
.
3
.
1

Installation

The Messaging library composes of one main jar fi
le and has
the
three dependencies
ZeroMQ
,
log4net and
ZeroMQ
binding for Java language. You can get the latest ve
rsions of these
libraries from
svn://first.ijs.si/ATOS/32/Library/Java
1

for 32bit
DLLs

and
svn://fi
rst.ijs.si/ATOS/64/Library/Java

for 64bit o
nes.

For the sake of this deliverable,
a current snapshot of the code and libraries can be obtained under:
http://project
-
first.eu/content/d71
-
integration
-
infrastructure
-
release
.

List of compiled artifacts:



messenger
-
0.0.1
-
SNAPSHOT.jar (Messaging library)



slf4j
-
log4j12
-
1.5.11.jar (Logging library)



log4j
-
1.2.16.jar (Logging library)



commons
-
collections
-
3.2.1.jar, commons
-
configuration
-
1.6.jar, commons
-
lang
-
2.4.jar,
commons
-
logging
-
1.1.1.jar (configuration library)



jzmq
-
2.1.0
-
SNAPSHOT.jar, libzmq.dll, jzmq.dl
l (
ZeroMQ
library for Java)






1

All the SVN
-
referred files and code can be also found in ZIP file attached to this deliverable.




D7.1


©
FIRST

consortium


Page
16

of
28

Steps for integrating

these libraries:

1
.

Add libzmq.dll and jzmq.dll to the System path or to the application directory.



Figure
6
: Setting up ZeroMQ libraries for Java implementation


2
.

Add the jar files located in the library folder to the project. For Maven projects, there is a
sample pom file that includes the required libraries to the projects.
svn://first.ijs.si/ATOS/Messenger Test JAVA/pom.xml
. Using Eclipse IDE, you
can include the
se libraries also from the project properties.






D7.1


©
FIRST

consortium


Page
17

of
28


Figure
7
: Adding ZeroMQ libraries to the Java project in Eclipse

2
.
3
.
2

Configuration

The Messaging library gets parameters via configuration files. Check out the resources

folder
to you
r project folder
svn://first.ijs.si/ATOS/64/Messenger JAVA Test/resources
. The
two configuration files should be located in the folder “resources
\
config” relative to the project
folder.

Config.xml
: Messaging parameters



ID: Id of the program to distinguish
from other components in the pipeline
.



ReceiverNumber: Number of components
that

send messages to the component
.



ProducerNumber: Number of components
that

produce messages in
parallel.



MessageSendAddress: Port to send messages
.



MessageReceiveAddress: Port

to receive messages
.



SendLoadBalancingAddress: Port to send load balancing commands to message
producer to stop receiving messages
.



ReceiveLoadBalancingAdress: Port to receive load balancing commands from message
consumer to stop sending messages.



FinishP
ublish: Port to publish
a
finish command to terminate messaging
.



FinishReceive: Port to listen

to the

finish command
.



MessagingType:

ZeroMQ

s
upports various messaging types:


o

0=Pipeline, 1=Request and Reply
.



MAX_QUEUE_SIZE: In

the

pipeline messaging type,
the
message sender doesn’t wait

for the

message receiver to consume these messages.
The m
essage sender stops
sending messages, if

the

queue size exceeds the max value.



MIN_QUEUE_SIZE: Minimum queue size to send continue message
s

to
the
message
producer
.




D7.1


©
FIRST

consortium


Page
18

of
28



IGNORE_QUEUE_OVERFLOW: Ignores messages when

a

data peak occurs.



Broker: When

the

receiver can’t consume all messages, messages can be sent to a
message
broker to consume them later on:


o

0=File storage, 1=Activemq
.



ACTIVEMQ: Activemq messaging port
.



QueueN
ame: Name of the queue used between WP3 and WP4
.



InFileStorageAddress: When messaging is terminated before consuming all messages,
messages are stored in files in this directory to consume later on
.



OutFileStorageAddress: When message send queue exceeds th
e maximum value or
messaging terminates before sending all messages to the consumer, messages are
stored in files in this directory.



maxFileStorageNum: Maximum number of messages that can be stored in files.



DBLoggingReceiver: Messaging port for logging me
ssag
es

in a database
.



DB_LOGGING: Enables logging messages in a database
.



log4net: Contains the logging configuration settings. You may change the level value
between “info and debug” to change logging details. Current configuration settings write
logging
information both in console and in a file (messenger.log).

Sample configuration file is presented in
Annex 2
: both main configuration and logging example.

2
.
3
.
3

API

for J
ava

Currently Messaging API contains one class Messenger. This class has one method open for
the messaging purposes.



String getMessage(): Returns the first received message in the queue. Waits until a
message is received.



Void sendMessage(String message):

Sends the input message file to message
consumers.



void stopMessaging(): Sends finish message to the message consumer and terminates
messaging.



bool isMessagingFinished(): Returns true if finish command is received from one of the
components in the pipeli
ne.

2
.
3
.
4

Sample code

There is

a

test project, which receives test messages from a test program. You may use this
project for testing purposes (
svn://first.ijs.si/ATOS/64/Messenger JAVA Test
).

The following code block receives test messages and sends messages t
o

the

message
consumers until
a
finish message is received.








D7.1


©
FIRST

consortium


Page
19

of
28

Messenger messenger = new Messenger();

while

(!Messenger.
isMessagingFinished
()) {


String message = messenger.getMessage();


if

(message !=
null
) {



logger
.info(
"Gets message
number:"

+ messageNum++);



messenger.sendMessage(message);


logger
.info(
"Sends message number:"

+ messageNum);


Thread.
sleep
(delay);



}

}

Listing
2
: Example of Java API





D7.1


©
FIRST

consortium


Page
20

of
28

3
.

Hardware resources

The FIRST pipeline is currently running on a single dedicated server machine. The machine has
25 terabytes of disk space plus 200 gigabytes of solid
-
state disk. It has 256 gigabytes of
memory and is able to run up to 48 concurrent processes (4 processors, 12 cores each),
running on 64bit Windows Server 2008 operating system. At this time, it is expected that this
machine will be enough to analyze the data, being acquired in FIRST, in rea
l time. However, if
these capacities prove insufficient, the integration middleware (ZeroMQ messaging technology)
will be used to distribute the data
-
analysis pipeline across several machines. Technically, this
means that the pipeline will end with a ZeroM
Q emitter at some point and continue with a
ZeroMQ receiver on a different machine. In this setting, the data will be processed to a certain
stage on one machine and then sent via the messaging "channel" to another machine in the
network for further proces
sing. In addition, it would be also possible to perform scaling
techniques (e.g. load balancing) across different machines as pointed out in

(FIRST D2.3
Scaling Strategy, 2011)
. Also other aspects
are considered,
such as instanti
ate processing
components dynamically (per demand) but we currently do not plan to implement these
complex scaling mechanisms in FIRST. Note that in reality, we do not run one single pipeline
but rather a workflow consisting of several pipelines. Even so,
the intuitions and principles for
distributed processing of FIRST data streams remain the same.
1




1

This paragraph is taken directly from
(FIRST D2.2 Conceptual and technical integrated architecture design, 2011)




D7.1


©
FIRST

consortium


Page
21

of
28

4
.

Conclusion


The integration component delivered as a part of this deliverable forms an important part of the
pipeline integration in the FIRST project.

This do
cument describes necessary configuration details for the pipeline middleware
component
s

that are

used for stream
-
based integration of technical components (such as
components of WP3 with WP4 and WP4 with WP6) according to the technical architecture
devised

within WP2.
Further on, this integration software is a part of
the
integrated FIRST
system, providing necessary data transport for the stream of processed documents.


Therefore this deliverable
is focused

on

pipeline
-
level integration software, while deliverable
D7.2
(FIRST D7.2 Early prototype of Integrated Financial Market Information system, 2012)

(and further WP7 deliverables in the future) describes

an

overall integrated versio
n of all
software components, including those
that

are not part of the FIRST pipeline
.







D7.1


©
FIRST

consortium


Page
22

of
28

References

FIRST D2.2 Conceptual and technical integrated architecture design. (2011).

FIRST D2.3 Scaling Strategy. (2011).

FIRST D7.2 Early prototype o
f Integrated Financial Market Information system. (2012).








D7.1


©
FIRST

consortium


Page
23

of
28

Annex
1
.

Example configuration file

for C# component


Unnecessary configuration fields (.NET specific) were removed for brevity.


<?
xml

version
=
"
1.0
"
?>

<
configuration
>


<
configSections
>


<
section

name
=
"
log4net
"

type
=
"
log4net.Config.Log4NetConfigurationSectionHandler, log4net
"

/>


</
configSections
>


<
appSettings
>


<
add

key
=
"
ID
"

value
=
"
1
"

/>


<
add

key
=
"
ReceiverNumber
"

value
=
"
1
"

/>


<
add

key
=
"
ProducerNumber
"

value
=
"
1
"

/>



<
add

key
=
"
MessageReceiveAddress
"

value
=
"
tcp://127.0.0.1:5555
"

/>


<
add

key
=
"
SendLoadBalancingAddress
"

value
=
"
tcp://127.0.0.1:5556
"

/>


<
add

key
=
"
FinishReceive
"

value
=
"
tcp://127.0.0.1:5550
"

/>


<
add

key
=
"
MessageSendAddress
"

value
=
"
tcp://127.0.0.1:5557
"

/>


<
add

key
=
"
ReceiveLoadBalancingAdress
"

value
=
"
tcp://127.0.0.1:5558
"

/>


<
add

key
=
"
FinishPublish
"

value
=
"
tcp://127.0.0.1:5550
"

/>


<!
--

0=Pipeline, 1=Request and Reply
--
>


<
add

key
=
"
MessagingType
"

value
=
"
0
"

/>


<
add

key
=
"
MAX_QUEUE_SIZE
"

value
=
"
5
"

/>


<
add

key
=
"
MIN_QUEUE_SIZE
"

value
=
"
1
"

/>


<
add

key
=
"
IGNORE_QUEUE_OVERFLOW
"

value
=
"
0
"

/>


<
add

key
=
"
Broker
"

value
=
"
0
"

/>


<
add

key
=
"
ACTIVEMQ
"

value
=
"
tcp://localhost:61616
"

/>


<
add

key
=
"
WAIT_COMMAND
"

value
=
"
WAIT
"

/>


<
add

key
=
"
FINISH_COMMAND
"

value
=
"
FINISH
"

/>


<
add

key
=
"
CONTINUE_COMMAND
"

value
=
"
CONTINUE
"

/>


<
add

key
=
"
MESSAGE_REQUEST
"

value
=
"
R
"

/>


<
add

key
=
"
RECEIVE_COMMAND_FILTER
"

value
=
""

/>


<
add

key
=
"
QueueName
"

value
=
"
WP3_QUEUE
"

/>


<
add

key
=
"
InFileStorageAddress
"

value
=
"
inMessageStorage
"

/>


<
add

key
=
"
OutFileStorageAddress
"

value
=
"
outMessageStorage
"

/>


<
add

key
=
"
MAX_FILE_STORAGE_SIZE
"

value
=
"
1000
"

/>


<
add

key
=
"
DBLoggingReceiver
"

value
=
"
tcp://*:5561
"

/>


<
add

key
=
"
DB_LOGGING
"

valu
e
=
"
false
"

/>


<
add

key
=
"
ClientSettingsProvider.ServiceUri
"

value
=
""

/>


</
appSettings
>


<
log4net
>


<
appender

name
=
"
LogFileAppender
"

type
=
"
log4net.Appender.FileAppender
"
>


<
param

name
=
"
File
"

value
=
"
logs/messenger.log
"

/>


<
param

name
=
"
AppendToFile
"

value
=
"
true
"

/>


<
layout

type
=
"
log4net.Layout.PatternLayout
"
>


<
param

name
=
"
Header
"

value
=
"
[Header]
\
r
\
n
"

/>


<
param

name
=
"
Footer
"

value
=
"
[Footer]
\
r
\
n
"

/>


<
param

name
=
"
ConversionPattern
"

value
=
"
%d [%t] %
-
5p %c %m%n
"

/>


</
layout
>


</
appender
>


<
appender

name
=
"
ConsoleAppender
"

type
=
"
log4net.Appender.ConsoleAppender
"
>


<
layout

type
=
"
log4net.Layout.PatternLayout
"
>


<
param

name
=
"
Header
"

value
=
"
[Header]
\
r
\
n
"

/>


<
param

name
=
"
Footer
"

value
=
"
[Footer]
\
r
\
n
"

/>


<
param

name
=
"
ConversionPattern
"

value
=
"
%d [%t] %
-
5p %c %m%n
"

/>


</
layout
>


</
appender
>


<
root
>


<
level

value
=
"
INFO
"

/>


<
appender
-
ref

ref
=
"
LogFileAppender
"

/>


<
appender
-
ref

ref
=
"
ConsoleAppender
"

/>


<
/
root
>


</
log4net
>

</
configuration
>





D7.1


©
FIRST

consortium


Page
24

of
28

Annex
2
.

Example configuration file for Java component


Annex a
.

Main configuration


<?
xml

version
=
"1.0"

encoding
=
"ISO
-
8859
-
1"

?>

<
messenger
-
definition
>


<
messaging
>


<
ID
>
1
</
ID
>


<
ReceiverNumber
>
1
</
ReceiverNumber
>


<
ProducerNumber
>
1
</
ProducerNumber
>


<
connection
>




<
MessageReceiveAddress
>
tcp://127.0.0.1:5557
</
MessageReceiveAddress
>



<
SendLoadBalancingAddress
>
tcp://127.0.0.1:5558
</
SendLoadBalancingAddress
>



<
FinishReceive
>
tcp://127.0.0.1:5550
</
FinishReceive
>



<
MessageSendAddress
>
tcp
://127.0.0.1:5555
</
MessageSendAddress
>



<
ReceiveLoadBalancingAddress
>
tcp
://127.0.0.1:5556
</
ReceiveLoadBalancingAddress
>

<
FinishPublish
>
tcp
://127.0.0.1:5550
</
FinishPublish
>



<
RECEIVE_COMMAND_FILTER
></
RECEIVE_COMMAND_FILTER
>



<
ACTIVEMQ
>
tcp
://127.0.0.1:61616
</
ACTIVEMQ
>



<
QueueName
>
WP3_QUEUE
</
QueueName
>


</
connection
>



<
MessagingType
>
0
</
MessagingType
>



<
Broker
>
0
</
Broker
>



<
MAX_QUEUE_SIZE
>
10
</
MAX_QUEUE_SIZE
>



<
MIN_QUEUE_SIZE
>
1
</
MIN_QUEUE_SIZE
>



<
BLOCKING_QUEUE
>
FALSE
</
BLOCKING_QUEUE
>



<
IGNORE_QUEUE_OVERFLOW
>
FALSE
</
IGNORE_QUEUE_OVERFLOW
>



<
MESSAGE_REQUEST
>
R
</
MESSAGE_REQUEST
>



<
WAIT_COMMAND
>
WAIT
</
WAIT_COMMAND
>



<
FINISH_COMMAND
>
FINISH
</
FINISH_COMMAND
>



<
CONTINUE_COMMAND
>
CONTINUE
</
CONTINUE_COMMAND
>



<
outFileStorageAddress
>
outMessageStorage
</
outFileStorageAddress
>



<
inFileStorageAddress
>
inMessageStorage
</
inFileStorageAddress
>



<
MAX_FILE_STORAGE_SIZE
>
1000
</
MAX_FILE_STORAGE_SIZE
>


</
messaging
>

</
messenger
-
definition
>


Annex b
.

Logging configuration


<?
xml

version
=
"1.0"

encoding
=
"UTF
-
8"
?>

<!
DOCTYPE

log4j:configuration

SYSTEM

"log4j.dtd"

>

<
log4j:configuration
>


<
appender

name
=
"stdout"

class
=
"org.apache.log4j.ConsoleAppender"
>


<
layout

class
=
"org.apache.log4j.PatternLayout"
>


<
param

name
=
"ConversionPattern"

value
=
"%d{
ABSOLUTE}


%5p %c{1}:%L
-

%m%n"
/>


</
layout
>


</
appender
>




<!
--

log all logs to a separate log file every day
--
>


<
appender

name
=
"orgDevelopersLogFileAppender"

class
=
"org.apache.log4j.DailyRollingFileAppender"
>


<
param

name
=
"file"

value
=
"messenger.log"

/>


<
param

name
=
"datePattern"

value
=
"'.'yyyy
-
MM
-
dd"

/>


<
param

name
=
"append"

value
=
"true"

/>


<
layout

class
=
"org.apache.log4j.PatternLayout"
>


<
param

name
=
"ConversionPattern"

value
=
"%d [%t] %
-
5p %C{6}
(%F:%L)
-

%m%n"
/>


</
layout
>




D7.1


©
FIRST

consortium


Page
25

of
28


</
appender
>


<
root
>



<
priority

value
=
"info"
></
priority
>



<
appender
-
ref

ref
=
"stdout"
/>



<
appender
-
ref

ref
=
"orgDevelopersLogFileAppender"

/>


</
root
>

</
log4j:configuration
>





D7.1


©
FIRST

consortium


Page
26

of
28

Annex
3
.

Configuration of the messaging test scenario


The following chapter contains
a
description of the testbed configuration of messaging
components, used during

the

component development and testing.

The runnable version can
be found on the FIRST machine under the following path: D:
\
ATOS
\
64
\
.


Annex a
.

Configurati
on of components

mimicking WP3
behaviour
(data emitters)


The following listing shows configuration of the testing scenario for WP3 components. The
configuration is the same for both data emitters.


<
configuration
>


<
appSettings
>


<
add

key
=
"
ID
"

value
=
"
1
"

/>


<
add

key
=
"
ReceiverNumber
"

value
=
"
4
"

/>


<
add

key
=
"
ProducerNumber
"

value
=
"
2
"

/>


<
add

key
=
"
MessageSendAddress
"

value
=
"
tcp://127.0.0.1:5555
"

/>


<
add

key
=
"
ReceiveLoadBalancingAdress
"

value
=
"
tcp://127.0.0.1:5561 tcp://127.0.0.1:5562



tcp://127.0.0.1:5563 tcp://127.0.0.1:5564
"

/>


<
add

key
=
"
FinishPublish
"

value
=
"
tcp://127.0.0.1:5550
"

/>


<!
--

0=Pipeline, 1=Request and Reply
--
>


<
add

key
=
"
MessagingType
"

value
=
"
0
"

/>


<
add

key
=
"
MAX_QUEUE_SIZE
"

value
=
"
5
"

/>


<
add

key
=
"
MIN_QUEUE_SIZE
"

value
=
"
1
"

/>


<
add

key
=
"
IGNORE_QUEUE_OVERFLOW
"

value
=
"
0
"

/>


<
add

key
=
"
Broker
"

value
=
"
0
"

/>


<
add

key
=
"
ACTIVEMQ
"

value
=
"
tcp://localhost:61616
"

/>


<
add

key
=
"
WAIT_COMMAND
"

value
=
"
WAIT
"

/>


<
add

key
=
"
FINISH_COMMAND
"

value
=
"
FINISH
"

/>


<
add

key
=
"
CONTINUE_COMMAND
"

value
=
"
CONTINUE
"

/>


<
add

key
=
"
MESSAGE_REQUEST
"

value
=
"
R
"

/>


<
add

key
=
"
RECEIVE_COMMAND_FILTER
"

value
=
""

/>


<
add

key
=
"
QueueName
"

value
=
"
WP3_QUEUE
"

/>


<
add

key
=
"
InFileStorageAddress
"

value
=
"
inMessageStorage
"

/>


<
add

key
=
"
OutFileStorageAddress
"

value
=
"
outMessageStorage
"

/>


<
add

key
=
"
MAX_FILE_STORAGE_SIZE
"

value
=
"
1000
"

/>


<
add

key
=
"
DBLoggingReceiver
"

value
=
"
tcp://*:5561
"

/>


<
add

key
=
"
DB_LOGGING
"

value
=
"
false
"

/>


<
add

key
=
"
ClientSettingsProvider.ServiceUri
"

value
=
""

/>


</
appSettings
>

<
/
configuration
>


Annex b
.

Configuration

for mimicking

WP4 behaviour

(data
receivers and emitters)


In the testbed scenario, there are four WP4 components. The following configuration

applies to
the first component:


<
messenger
-
definition
>


<
messaging
>


<
ID
>
1
</
ID
>


<
ReceiverNumber
>
2
</
ReceiverNumber
>


<
ProducerNumber
>
4
</
ProducerNumber
>


<
connection
>




D7.1


©
FIRST

consortium


Page
27

of
28



<
MessageReceiveAddress
>
tcp
://127.0.0.1:5555


tcp
://127.0.0.1:5556
</
MessageReceiveAddress
>



<
SendLoadBalancingAddress
>
tcp
://127.0.0.1:5561
</
SendLoadBalancingAddress
>



<
FinishReceive
>
tcp
://127.0.0.1:5550
</
FinishReceive
>



<
MessageSendAddress
>
tcp
://127.0.0.1:5557
</
MessageSendAddress
>



<
ReceiveLoadBalancingAddress
>
tcp
://127.0.0.1:5565



tcp
://127.0.0.1:5566
</
ReceiveLoadBalancingAddress
>


<
RECEIVE_COMMAND_FILTER
></
RECEIVE_COMMAND_FILTER
>


<
ACTIVEMQ
>
tcp
://127.0.0.1:61616
</
ACTIVEMQ
>


<
QueueName
>
WP3_QUEUE
</
QueueName
>


</
connection
>


<
MessagingType
>
0
</
MessagingType
>


<
Broker
>
0
</
Broker
>


<
MAX_QUEUE_SIZE
>
10
</
MAX_QUEUE_SIZE
>


<
MIN_QUEUE_SIZE
>
1
</
MIN_QUEUE_SIZE
>


<
BLOCKING_QUEUE
>
FALSE
</
BLOCKING_QUEUE
>


<
IGNORE_QUEUE_OVERFLOW
>
FALSE
</
IGNORE_QUEUE_OVERFLOW
>


<
MESSAGE_REQUEST
>
R
</
MESSAGE_REQUEST
>


<
WAIT_COMMAND
>
WAIT
</
WAIT_COMMAND
>


<
FINISH_COMMAND
>
FINISH
</
FINISH_COMMAND
>


<
CONTINUE_COMMAND
>
CONTINUE
</
CONTINUE_COMMAND
>


<
outFileStorageAddress
>
outMessageStorage
</
outFileStorageAddress
>


<
inFileStorageAddress
>
inMessageStorage
</
inFileStorageAddress
>


<
MAX_FILE_STORAGE_SIZE
>
1000
</
MAX_FILE_STORAGE_SIZE
>


</
messaging
>

</
messenger
-
definition
>


Second c
omponent

(
showing
only differing lines
)
:

<
SendLoadBalancingAddress
>
tcp
://127.0.0.1:5562
</
SendLoadBalancingAddress
>

<
MessageSendAddress
>
tcp
://127.0.0.1:5558
</
MessageSendAddress
>


Third component (
showing
only d
iffering lines
):

<
SendLoadBalancingAddress
>
tcp
://127.0.0.1:5563
</
SendLoadBalancingAddress
>

<
MessageSendAddress
>
tcp
://127.0.0.1:5559
</
MessageSendAddress
>


Fourth comoponent (
showing
only differing lines
):

<
SendLoadBalancingAddress
>
tcp
://127.0.0.1:5564
</
SendLoadBalancingAddress
>

<
MessageSendAddress
>
tcp
://127.0.0.1:5560
<
/
MessageSendAddress
>















D7.1


©
FIRST

consortium


Page
28

of
28

Annex c
.

Configuration
for

mimicking WP6 behaviour (data
receivers only)


The following listing shows the configuration

of the first of two WP6 components
.


<
configuration
>


<
appSettings
>


<
add

key
=
"
ID
"

value
=
"
1
"

/>


<
add

key
=
"
ReceiverNumber
"

value
=
"
2
"

/>


<
add

key
=
"
ProducerNumber
"

value
=
"
4
"

/>


<
add

key
=
"
MessageReceiveAddress
"

value
=
"
tcp://127.0.0.1:5557 tcp://127.0.0.1:5558
tcp://127.0.0.1:5559 tcp://127.0.0.1:5560
"

/>


<
add

key
=
"
SendLoadBalancingAddress
"

value
=
"
tcp://127.0.0.1:5565
"

/>


<
add

key
=
"
FinishReceive
"

value
=
"
tcp://127.0.0.1:5550
"

/>


<!
--

0=Pipeline, 1=Request and Reply
--
>


<
add

key
=
"
MessagingType
"

value
=
"
0
"

/>


<
add

key
=
"
MAX_QUEUE_SIZE
"

value
=
"
5
"

/>


<
add

key
=
"
MIN_QUEUE_SIZE
"

value
=
"
1
"

/
>


<
add

key
=
"
IGNORE_QUEUE_OVERFLOW
"

value
=
"
0
"

/>


<
add

key
=
"
Broker
"

value
=
"
0
"

/>


<
add

key
=
"
ACTIVEMQ
"

value
=
"
tcp://localhost:61616
"

/>


<
add

key
=
"
WAIT_COMMAND
"

value
=
"
WAIT
"

/>


<
add

key
=
"
FINISH_COMMAND
"

value
=
"
FINISH
"

/>


<
add

key
=
"
CONTINUE_COMMAND
"

value
=
"
CONTINUE
"

/>


<
add

key
=
"
MESSAGE_REQUEST
"

value
=
"
R
"

/>


<
add

key
=
"
RECEIVE_COMMAND_FILTER
"

value
=
""

/>


<
add

key
=
"
QueueName
"

value
=
"
WP3_QUEUE
"

/>


<
add

key
=
"
InFileStorageAddress
"

value
=
"
inMessageStorage
"

/>


<
add

key
=
"
OutFileStorageAddress
"

value
=
"
outMessageStorage
"

/>


<
add

key
=
"
MAX_FILE_STORAGE_SIZE
"

value
=
"
1000
"

/>


<
add

key
=
"
DBLoggingReceiver
"

value
=
"
tcp://*:5561
"

/>


<
add

key
=
"
DB_LOGGING
"

value
=
"
false
"

/>


<
add

key
=
"
ClientSettingsProvider.ServiceUri
"

value
=
""

/>


</
appSettings
>

</
configuration
>


The
configuration of the second components

is the same except for the following line:

<
add

key
=
"
SendLoadBalancingAddress
"

value
=
"
tcp://127.0.0.1:5566
"

/>