Project Acronym:
FIRST
Project Title:
Large scale information extraction and
integration infrastructure for supporting
financial decision making
Project Number:
257928
Instrument:
STREP
Thematic Priority:
ICT
-
2009
-
4.3 Information and Communication
Technology
D
7.1
Integration
Infrastructure Release
Work Package:
WP
7
–
Integrated Financial Market Information
System
Due Date:
31
/
03
/
2012
Submission Date:
3
0/03/2012
Start Date of
Project:
01/10/2010
Duration of Project:
36 Months
Organisation Responsible
for
Deliverable:
ATOS
Version:
1.0
Status:
Final
Author
(s)
:
Mateusz Radzimski
Murat Kalender
ATOS
ATOS
Reviewer(s):
Irina Alic
Julia Zillies
UGOE
UGOE
Nature:
R
–
Report
P
–
Prototype
D
–
Demonstrator
O
-
Other
Dissemination level:
PU
-
Public
CO
-
Confidential, only for members of the
consortium (including the Commission)
RE
-
Restricted to a group specified by the
consortium (including the Commission Services)
Project co
-
funded by the European Commission within the Seventh Framework Programme (2007
-
2013)
D7.1
Revision history
Version
Date
Modified by
Comments
0.1
20
/
0
2
/
201
2
Mateusz Radzimski
(ATOS)
First
version of ToC
provided
0.2
20/02/2012
Mateusz Radzimski,
Murat Kalender
(ATOS)
Description of
the
integration component
0.3
2/03/2012
Mateusz Radzimski
Update of “Implementation
details” section.
0.4
5/03/2012
Mateusz Radzimski
Executive summary and
Conclusions
, Annex 1
1.0
30/03/2012
Mateusz Radzimski
Addressing reviewers’
comments
. Final formatting.
D7.1
Copyright © 201
2
,
FIRST Consortium
The FIRST Consortium (
www.project
-
first.eu
) grants third parties the right to use and distribute
all or parts of this document, provided that the FIRST project and the document are properly
referenced.
THIS DOCUMENT IS PROVIDED BY THE COPY
RIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTOR
S BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY T
HEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
DOCUMENT, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
----------------
D7.1
Executive Summary
Based on the
d
esign and simulation studies c
onducted throughout
WP2
, implementations of the
integration infrastructure prototypes have been started. This deliverable presents technical
details of the lightweight messaging middleware component used for FIRST pipeline int
egration.
It is in line with the conceptual and technical design and the scaling strategy (see
(FIRST D2.2
Conceptual and technical integrated architecture design, 2011)
and
(FIRST D2.3 Scaling
Strategy, 201
1)
).
This deliverable is accompanied by the software prototype that
realizes
stream
-
based
integration
. The main objective is to provide
a
necessary technical description for configuring
and running the software component.
This software is also being
used by the early integrated version of FIRST components
(FIRST
D7.2 Early prototype of Integrated Financial Market Information system, 2012)
and may likely
be
subject to minor improvements
within the development and integration
of the overall FIRST
inf
rastructure.
O
ther aspects of FIRST integration are subject of the de
liverable D7.2 and future
WP7 deliverables.
D7.1
Table of Contents
Executive Summary
................................
................................
................................
......
4
Abbreviations and acronyms
................................
................................
.......................
7
1.
Introduction
................................
................................
................................
............
8
1.1.
Purpose of this document
................................
................................
..................
8
2.
Lightweight integration component
................................
................................
......
9
2.1.
Implementation details
................................
................................
......................
9
2.2.
C# Integration
................................
................................
................................
..
11
2.2.1
Instal
lation
................................
................................
................................
11
2.2.2
Configuration
................................
................................
............................
13
2.2.3
API for C#
................................
................................
................................
.
14
2.2.4
Sample code
................................
................................
............................
15
2.3.
Java Inte
gration
................................
................................
...............................
15
2.3.1
Installation
................................
................................
................................
15
2.3.2
Configuration
................................
................................
............................
17
2.3.3
API for Java
................................
................................
..............................
18
2.3.4
Sample code
................................
................................
............................
18
3.
Hardware resources
................................
................................
.............................
20
4.
Conclusion
................................
................................
................................
............
21
References
................................
................................
................................
...................
22
Annex 1.
Example configuration file for C# component
................................
.....
23
Annex 2.
Example configuration file for Java component
................................
.
24
Annex a.
Main configuration
................................
................................
....................
24
Anne
x b.
Logging configuration
................................
................................
...............
24
Annex 3.
Configuration of the messaging test scenario
................................
....
26
Annex a.
Configuration of components mimicking WP3 behaviour (data emitters)
.
26
Annex b.
Configuration for mimicking WP4 behaviour (data receivers and emitters)
26
Annex c.
Configuration for mimicking WP6 behaviour (data receivers only)
...........
28
Index of Figures
Figure 1:
Technical details of integration component
................................
................................
.....
9
Figure 2: Sample load balancing setup (2x and 1x mean component’s throughput as a numb
er of
messages processed at once).
................................
................................
................................
........
10
Figure 3: Setting up ZeroMQ libraries for C# implementation
................................
....................
12
Figure 4: Adding ZeroMQ references to the C# project in Visual Studio
................................
....
12
Figure 5: Adding ZeroMQ libraries to the C# project in Visual Studio (2)
................................
..
13
Figure 6: Setting up ZeroMQ libraries for Java implementation
................................
..................
16
Figure 7: Adding ZeroMQ libraries to the Java project in Eclipse
................................
...............
17
D7.1
Index of
Listings
Listing 1: Example of C#
API
................................
................................
................................
.......
15
Listing 2: Example of Java API
................................
................................
................................
....
19
D7.1
©
FIRST
consortium
Page
7
of
28
Abbreviations and acronyms
IFMIS
Integrated Financial Market Information System
WP
Workpackage
ID
Identifier
DB
Datab
ase
DLL
Dynamic Link Library
API
Application Programming
Interface
JVM
Java Virtual Machine
D7.1
©
FIRST
consortium
Page
8
of
28
1
.
Introduction
Disclaimer: for reader’s convenience some fragments of this deliverable are repeated from WP2
deliverables.
1
.
1
.
Purpose of this document
This document presents
implementation and configuration details
of the integration
c
omponent
developed withi
n WP7.
The messaging component is implemented according to the design
guidelines from
(FIRST D2.2 Conceptual and technical integrated architecture design, 2011)
and is supporting scaling techniques as described in
(FIRST D2.3 Scaling Strategy, 2011)
.
We are also presenting a testing environment that mimics the pipeline configuration, for
messaging testing purposes.
Furthermore, t
his document is
also
accompanied by the
prototype
implementation source code
.
Please note that
t
he goal of this document is to present an isolated pipeline integration
approach. For a view of the current integrated set of components (where messaging
components plays an important role in the integration), see
(FIRST D7.2 Early prototype of
Integrated Financial Market Information system, 2012)
.
In the further
chapter
we
introduce the integration component and provide
technical details
of
implementation
and setup.
It will also cover
API
binding for both Java and C#. Next,
we
describe short overview of the
already running
deployment platform
.
The last chapter presents
conclusions from development of the integration component.
All configuration examples have
been put in annexes and a
re ref
erred from within the deliverable
.
D7.1
©
FIRST
consortium
Page
9
of
28
2
.
Lightweight integration
component
This chapter explains the implementation of the integration component
that provides
facilities for
pipeline components integration.
As the result of
the simulation studies accomplished within
WP2 and described in
(FIRST D2.2 Conceptual and technical integrated architecture design,
2011)
, we follow the messaging approach
as a way to process data streams through pipeline
components
.
The messaging approach adopted in FIRST can be simply explained as
data (message)
exchanging between components. T
echnically
it
means that
components in the FIRST pipeline
are
passing
messages
(i.e. web documents)
between each other
. They are
streamlined
so
that
dataf
low throughput c
an be optimal and no single component needs to orchestrate the whole
process. Rather each component works
individually by processing received messages and
forwardin
g results to the next component in line.
Th
e following subsections present
implementation and configuration details.
2
.
1
.
Implementation details
The messaging component has been implemented
based on
the
ZeroMQ
1
library.
The
design
and
performanc
e
considerations
have
been
carried on
in
(FIRST D2.2 Conceptual and technical
integrated architecture design, 2011)
.
This chapter describes
the
current implementation
features
of this component
from
a
practical point of view, as it is used
to
interconnect
p
ipeline
components.
Since pipeline components are developed by different partners using different technologies (
e.g.
WP3 components using .N
ET platform and WP4 using Java)
the messaging component needs
proper API
binding
s
to support those lan
guages
.
Therefore
the
component has
been developed
in two versions: C# and Java, having
the same functio
nalities
and corresponding API
for each
language.
The single component can be configured to be either emitter or receiver
of messages
or both
.
The
ZeroMQ library
is system
-
dependent and therefore it must be compiled for different
operating system
s
. In
the current
case it
has been compiled for both 32
-
bit and 64
-
bit platforms
.
Figure
1
: Technical details of integration comp
onent
2
The most important feature
i
s
the
fast exchange of messages between two (or more)
messaging components.
It currently supports
both
messaging scenarios
that can be configured
using
configuration file (
for details
see
paragraphs
2.2.2
for C# and
2.3.2
for Java
)
:
1
ZeroMQ library: http://www.zeromq.org/
2
As designed in
(FIRST D2.2 Conceptual and technical integrated architecture design, 2011)
D7.1
©
FIRST
consortium
Page
10
of
28
Request/Reply
:
a
classical
scenario
where
sender and receiver exchange messages
one by one in a blocking fash
ion
where
each message is confirmed before the next one
is sent.
Pipel
ine
: messages are pushed in a stream
-
based manner from
the
emitter to the
receiver.
Messages are sent asynchronously,
without
an
explicit request from the
message consumer
.
On the receiver part, messages are kept in a queue
before they are
consumed.
This a
pproach performs much faster than Request/Reply, but at a cost of
additional
synchronisation
and
this approach
pose
s
a
risks of queue overflow
(and
message dropping)
when too big message peak
s
occurs
.
In a simple scenario, overflow
is handled by using
an
additional control channel
between
the
message emitter and
the
message receiver
to notify
the queue state
.
In addition
to that,
the
mess
a
ging component
support
s
multiple senders and receivers and in
such case
s
load balancing between the nodes is performed
according to
FIRST D2.3
deliverable
(FIRST D2.3 Scaling Strategy, 2011)
as shown in a simplified version with two
receivers on
Figure
1
.
L
oad balancing aims at directing messages to the least occupied
component, thus distributing overall processing load as evenly as possible.
More
technical
details have been exposed in D2.3
deliverable
.
Figure
2
:
Sample load
balancing setup
(
2
x and 1x mean
component’s throughput as a
n
umber
of messages
process
ed
at once)
.
Figure
2
presents a more complex test scenario setup
of
the
compon
ents
for load balancing
experiments.
Messages are produced within two WP3 components, and are sent to four WP4
consumers. At this moment
a
load balancing algorithm performs a distribution of messages
between all receiving components.
When there are multipl
e consumers and producers in the
pipeline, we need a scheduling algorithm. We assume that message consumers are identical
and consume
a
similar amount of messages at a time.
If
the queue of a message consumer is
full, an “overflow” message is sent through
the control channel.
All “overflow” messages are
sent to each message producer therefore every producer knows how many consumers have
their queue full.
The flow of the messages is controlled
by
using IDs and consumer/producer
lists.
In
the
test scenario
(
two
WP3
producer
s
and
four
WP4
consumer
s)
if one
consumer sends
an
overflow message all producers
will
continue messaging.
If the
second
overflow message is
sent,
the
producer with ID value 1
will
stop sending messages.
After WP4 finishes
processing
messag
es
, results are sent up the pipeline to
the
WP6 components. In this scenario
WP6
components
perform as a “sink” receiving messages from multiple producers. Configuration
details of this sample scenario are presented in
Annex 1
.
Please note that
the
presented
scenario is for messaging middleware
experimenting
testbed
only
and is not the exact target
FIRST deployment scenario
.
D7.1
©
FIRST
consortium
Page
11
of
28
Other implemented features of the messagin
g component:
Handling failures:
T
o prevent data loss the messaging component is able to redirect all
overflowing messages in queues to
a
persistent storage
. In the target FIRST deployment
scenario this feature will not be used, as the pipeline will be dime
nsioned according to
the traffic to be processed.
Multithreading
:
The
messaging library has been multithreaded for performance reasons.
We use separate threads for sending
and receiving messages
and for control channel
messaging
.
Queue data structure
:
O
n
the sender and receiver, all messages are store
d
in a
FIFO
queue
s
.
Logging:
The
component provides logging through the external logging library (Log4j for
Java and log4net for C# implementation)
2
.
2
.
C
# Integration
The messaging component provides
an
implement
ation endpoint (emitter
and
receiver) for use
with
a
C# platform.
The f
ollowing
sub
sections explain
the
configuration and installation details of
the C# implementation.
2
.
2
.
1
Installation
The Messaging library composes of one main
DLL
file and ha
s
the following
three
dependencies
:
Z
ero
MQ
, log4net and
ZeroMQ
binding for C# language. You can get the latest
versions of these libraries from
svn://first.ijs.si/ATOS/32/library/C#
1
for 32bit
DLLs
and
svn://first.ijs.si/ATOS/64
for 64bit
DLLs
.
For the sake of this deliverable, a current
snapshot of the code and libraries can be obtained under:
http://project
-
first.eu/content/d71
-
integration
-
infrastructure
-
release
.
List of compiled artifacts:
Messenger.dll (Messaging library)
log4net.dll (Logging
library for C#)
libzmq.dll (
ZeroMQ
library)
clrzmq.dll (
ZeroMQ
library for C#)
1
All the
SVN
-
referred files and code can be also found in ZIP file attached
to this deliverable.
D7.1
©
FIRST
consortium
Page
12
of
28
To integrate these libraries in Visual Studio:
1
.
Add libzmq.dll to the System path or to main programs’ directory (Executable).
Figure
3
: Setting up ZeroMQ libraries for C# implementation
2
.
Right click on the References in Solution Explorer window and select “Add Reference”
Figure
4
: Adding ZeroMQ
references
to the C# project
in Visual Studio
D7.1
©
FIRST
consortium
Page
13
of
28
3
.
Click the Browse t
ab and select the checked out 3
DLL
files
Figure
5
: Adding ZeroMQ libraries to the C# project
in Visual Studio
(2)
2
.
2
.
2
Configuration
The Messaging library gets parameters via configuration files. The following parameters must
be added to
the
app.config file in order to use
the
messaging library:
ID: Id of the program to distinguish from other components in the pipeline
.
ReceiverNumber:
Number of components
that
send messages to the component
.
ProducerNumber: Number of components
that
produce messages in parallel
.
MessageSendAddress:
Port to send messages
.
MessageReceiveAddress: Port to receive messages
.
SendLoadBalancingAddress: Port to
send load balancing commands to
the
message
producer to stop receiving messages
.
ReceiveLoadBalancingAdress: Port to receive load balancing commands from
the
message consumer to stop sending messages.
FinishPublish: Port to publish
the
finish command to t
erminate messaging
.
FinishReceive: Port to listen
to the
finish command
.
MessagingType:
S
upport for various messaging types:
o
0=Pipeline, 1=Request and Reply
.
MAX_QUEUE_SIZE: In pipeline messaging type,
the
message sender doesn’t wait
for
the
message receiver to consume these messages.
The m
essage sender stops sending
messages, if
the
queue size exceeds the max value.
D7.1
©
FIRST
consortium
Page
14
of
28
MIN_QUEUE_SIZE: Minimum queue size to send continue message
s
to
the
message
producer
.
IGNORE_QUEUE_OVERFLOW: Ignores messages wh
en
a
data peak occurs.
Broker:
If the
receiver can’t consume all messages, messages can be sent to a message
b
roker to consume them later on
:
o
0=File storage, 1=Active
MQ
1
ACTIVEMQ: Active
MQ
messaging port
.
QueueName: Name of the queue used between WP3 and W
P4
.
InFileStorageAddress: When messaging is terminated before
all messages are
consumed
,
the
messages are stored in files in this directory to consume later on
.
OutFileStorageAddress:
If the
message send queue exceeds the maximum value or
messaging termina
tes before sending all messages to the consumer, messages are
stored in files in this directory.
maxFileStorageNum: Maximum number of messages that can be stored in files.
DBLoggingReceiver: Messaging port for logging messaging in a database
.
DB_LOGGING: E
nables logging messages in a database
.
log4net: Contains the logging configuration settings. You may change the level value
between “info and debug” to change logging details. Current configuration settings write
logging information both in console and in
a file (messenger.log).
You can use
the app.config file located under following
locations:
svn://first.ijs.si/ATOS/64
/Messenger C#/Test
/app.config
svn://first.ijs.si/ATOS/64
/Wp3 Test/Test.exe.config
svn://first.ijs.si/AT
OS/64/Wp6 Test/Test.exe.config
The sample configuration file used by
the
Messanger component
is presented in
Annex 1
.
2
.
2
.
3
API
for C#
Currently Messaging API contains one class Messenger. This class
has three methods open for
the messaging purposes:
void sendMessage(String message)
: Sends
an
input message to the message
consumer
String getMessage()
: Returns one message file from
the
received messages queue.
void stopMessaging()
: Sends
a
finish message
to the message consumer and
terminates messaging.
bool isMessagingFinished()
: Returns true if
the
finish command is received from one of
the components in the pipeline.
1
ActiveMQ
broker
-
based messaging system:
http://activemq.apache.org/
D7.1
©
FIRST
consortium
Page
15
of
28
2
.
2
.
4
Sample code
There is
a
test project, which sends test messages to the message
listener. You may use this
project for testing purposes (
svn://first.
ijs.si/ATOS/Messenger C#/Test
).
The following code block sends received messages
to message listener components:
Messenger
messenger =
new
Messenger
();
while
(!
messenger.isMessagingFinished())
{
String
inMessage = messenger.getMessage();
if
(inMessage !=
null
)
{
logger.Info(
"Gets message number: "
+ inMessageNum++);
messenger.sendMessage(inMessage);
logger.Info(
"Sends m
essage number: "
+ inMessageNum);
Thread
.Sleep(delay);
}
}
Listing
1
: Example of C# API
2
.
3
.
Java Integration
The second API implementation is the Java binding. The messaging component provides an
implementation endpoint
(emitter and receiver) for use with a JVM
(Java)
. The following
subsections explain the configuration and installation details of the
Java
implementation
and it is
analogical to the C# version.
2
.
3
.
1
Installation
The Messaging library composes of one main jar fi
le and has
the
three dependencies
ZeroMQ
,
log4net and
ZeroMQ
binding for Java language. You can get the latest ve
rsions of these
libraries from
svn://first.ijs.si/ATOS/32/Library/Java
1
for 32bit
DLLs
and
svn://fi
rst.ijs.si/ATOS/64/Library/Java
for 64bit o
nes.
For the sake of this deliverable,
a current snapshot of the code and libraries can be obtained under:
http://project
-
first.eu/content/d71
-
integration
-
infrastructure
-
release
.
List of compiled artifacts:
messenger
-
0.0.1
-
SNAPSHOT.jar (Messaging library)
slf4j
-
log4j12
-
1.5.11.jar (Logging library)
log4j
-
1.2.16.jar (Logging library)
commons
-
collections
-
3.2.1.jar, commons
-
configuration
-
1.6.jar, commons
-
lang
-
2.4.jar,
commons
-
logging
-
1.1.1.jar (configuration library)
jzmq
-
2.1.0
-
SNAPSHOT.jar, libzmq.dll, jzmq.dl
l (
ZeroMQ
library for Java)
1
All the SVN
-
referred files and code can be also found in ZIP file attached to this deliverable.
D7.1
©
FIRST
consortium
Page
16
of
28
Steps for integrating
these libraries:
1
.
Add libzmq.dll and jzmq.dll to the System path or to the application directory.
Figure
6
: Setting up ZeroMQ libraries for Java implementation
2
.
Add the jar files located in the library folder to the project. For Maven projects, there is a
sample pom file that includes the required libraries to the projects.
svn://first.ijs.si/ATOS/Messenger Test JAVA/pom.xml
. Using Eclipse IDE, you
can include the
se libraries also from the project properties.
D7.1
©
FIRST
consortium
Page
17
of
28
Figure
7
: Adding ZeroMQ libraries to the Java project in Eclipse
2
.
3
.
2
Configuration
The Messaging library gets parameters via configuration files. Check out the resources
folder
to you
r project folder
svn://first.ijs.si/ATOS/64/Messenger JAVA Test/resources
. The
two configuration files should be located in the folder “resources
\
config” relative to the project
folder.
Config.xml
: Messaging parameters
ID: Id of the program to distinguish
from other components in the pipeline
.
ReceiverNumber: Number of components
that
send messages to the component
.
ProducerNumber: Number of components
that
produce messages in
parallel.
MessageSendAddress: Port to send messages
.
MessageReceiveAddress: Port
to receive messages
.
SendLoadBalancingAddress: Port to send load balancing commands to message
producer to stop receiving messages
.
ReceiveLoadBalancingAdress: Port to receive load balancing commands from message
consumer to stop sending messages.
FinishP
ublish: Port to publish
a
finish command to terminate messaging
.
FinishReceive: Port to listen
to the
finish command
.
MessagingType:
ZeroMQ
s
upports various messaging types:
o
0=Pipeline, 1=Request and Reply
.
MAX_QUEUE_SIZE: In
the
pipeline messaging type,
the
message sender doesn’t wait
for the
message receiver to consume these messages.
The m
essage sender stops
sending messages, if
the
queue size exceeds the max value.
MIN_QUEUE_SIZE: Minimum queue size to send continue message
s
to
the
message
producer
.
D7.1
©
FIRST
consortium
Page
18
of
28
IGNORE_QUEUE_OVERFLOW: Ignores messages when
a
data peak occurs.
Broker: When
the
receiver can’t consume all messages, messages can be sent to a
message
broker to consume them later on:
o
0=File storage, 1=Activemq
.
ACTIVEMQ: Activemq messaging port
.
QueueN
ame: Name of the queue used between WP3 and WP4
.
InFileStorageAddress: When messaging is terminated before consuming all messages,
messages are stored in files in this directory to consume later on
.
OutFileStorageAddress: When message send queue exceeds th
e maximum value or
messaging terminates before sending all messages to the consumer, messages are
stored in files in this directory.
maxFileStorageNum: Maximum number of messages that can be stored in files.
DBLoggingReceiver: Messaging port for logging me
ssag
es
in a database
.
DB_LOGGING: Enables logging messages in a database
.
log4net: Contains the logging configuration settings. You may change the level value
between “info and debug” to change logging details. Current configuration settings write
logging
information both in console and in a file (messenger.log).
Sample configuration file is presented in
Annex 2
: both main configuration and logging example.
2
.
3
.
3
API
for J
ava
Currently Messaging API contains one class Messenger. This class has one method open for
the messaging purposes.
String getMessage(): Returns the first received message in the queue. Waits until a
message is received.
Void sendMessage(String message):
Sends the input message file to message
consumers.
void stopMessaging(): Sends finish message to the message consumer and terminates
messaging.
bool isMessagingFinished(): Returns true if finish command is received from one of the
components in the pipeli
ne.
2
.
3
.
4
Sample code
There is
a
test project, which receives test messages from a test program. You may use this
project for testing purposes (
svn://first.ijs.si/ATOS/64/Messenger JAVA Test
).
The following code block receives test messages and sends messages t
o
the
message
consumers until
a
finish message is received.
D7.1
©
FIRST
consortium
Page
19
of
28
Messenger messenger = new Messenger();
while
(!Messenger.
isMessagingFinished
()) {
String message = messenger.getMessage();
if
(message !=
null
) {
logger
.info(
"Gets message
number:"
+ messageNum++);
messenger.sendMessage(message);
logger
.info(
"Sends message number:"
+ messageNum);
Thread.
sleep
(delay);
}
}
Listing
2
: Example of Java API
D7.1
©
FIRST
consortium
Page
20
of
28
3
.
Hardware resources
The FIRST pipeline is currently running on a single dedicated server machine. The machine has
25 terabytes of disk space plus 200 gigabytes of solid
-
state disk. It has 256 gigabytes of
memory and is able to run up to 48 concurrent processes (4 processors, 12 cores each),
running on 64bit Windows Server 2008 operating system. At this time, it is expected that this
machine will be enough to analyze the data, being acquired in FIRST, in rea
l time. However, if
these capacities prove insufficient, the integration middleware (ZeroMQ messaging technology)
will be used to distribute the data
-
analysis pipeline across several machines. Technically, this
means that the pipeline will end with a ZeroM
Q emitter at some point and continue with a
ZeroMQ receiver on a different machine. In this setting, the data will be processed to a certain
stage on one machine and then sent via the messaging "channel" to another machine in the
network for further proces
sing. In addition, it would be also possible to perform scaling
techniques (e.g. load balancing) across different machines as pointed out in
(FIRST D2.3
Scaling Strategy, 2011)
. Also other aspects
are considered,
such as instanti
ate processing
components dynamically (per demand) but we currently do not plan to implement these
complex scaling mechanisms in FIRST. Note that in reality, we do not run one single pipeline
but rather a workflow consisting of several pipelines. Even so,
the intuitions and principles for
distributed processing of FIRST data streams remain the same.
1
1
This paragraph is taken directly from
(FIRST D2.2 Conceptual and technical integrated architecture design, 2011)
D7.1
©
FIRST
consortium
Page
21
of
28
4
.
Conclusion
The integration component delivered as a part of this deliverable forms an important part of the
pipeline integration in the FIRST project.
This do
cument describes necessary configuration details for the pipeline middleware
component
s
that are
used for stream
-
based integration of technical components (such as
components of WP3 with WP4 and WP4 with WP6) according to the technical architecture
devised
within WP2.
Further on, this integration software is a part of
the
integrated FIRST
system, providing necessary data transport for the stream of processed documents.
Therefore this deliverable
is focused
on
pipeline
-
level integration software, while deliverable
D7.2
(FIRST D7.2 Early prototype of Integrated Financial Market Information system, 2012)
(and further WP7 deliverables in the future) describes
an
overall integrated versio
n of all
software components, including those
that
are not part of the FIRST pipeline
.
D7.1
©
FIRST
consortium
Page
22
of
28
References
FIRST D2.2 Conceptual and technical integrated architecture design. (2011).
FIRST D2.3 Scaling Strategy. (2011).
FIRST D7.2 Early prototype o
f Integrated Financial Market Information system. (2012).
D7.1
©
FIRST
consortium
Page
23
of
28
Annex
1
.
Example configuration file
for C# component
Unnecessary configuration fields (.NET specific) were removed for brevity.
<?
xml
version
=
"
1.0
"
?>
<
configuration
>
<
configSections
>
<
section
name
=
"
log4net
"
type
=
"
log4net.Config.Log4NetConfigurationSectionHandler, log4net
"
/>
</
configSections
>
<
appSettings
>
<
add
key
=
"
ID
"
value
=
"
1
"
/>
<
add
key
=
"
ReceiverNumber
"
value
=
"
1
"
/>
<
add
key
=
"
ProducerNumber
"
value
=
"
1
"
/>
<
add
key
=
"
MessageReceiveAddress
"
value
=
"
tcp://127.0.0.1:5555
"
/>
<
add
key
=
"
SendLoadBalancingAddress
"
value
=
"
tcp://127.0.0.1:5556
"
/>
<
add
key
=
"
FinishReceive
"
value
=
"
tcp://127.0.0.1:5550
"
/>
<
add
key
=
"
MessageSendAddress
"
value
=
"
tcp://127.0.0.1:5557
"
/>
<
add
key
=
"
ReceiveLoadBalancingAdress
"
value
=
"
tcp://127.0.0.1:5558
"
/>
<
add
key
=
"
FinishPublish
"
value
=
"
tcp://127.0.0.1:5550
"
/>
<!
--
0=Pipeline, 1=Request and Reply
--
>
<
add
key
=
"
MessagingType
"
value
=
"
0
"
/>
<
add
key
=
"
MAX_QUEUE_SIZE
"
value
=
"
5
"
/>
<
add
key
=
"
MIN_QUEUE_SIZE
"
value
=
"
1
"
/>
<
add
key
=
"
IGNORE_QUEUE_OVERFLOW
"
value
=
"
0
"
/>
<
add
key
=
"
Broker
"
value
=
"
0
"
/>
<
add
key
=
"
ACTIVEMQ
"
value
=
"
tcp://localhost:61616
"
/>
<
add
key
=
"
WAIT_COMMAND
"
value
=
"
WAIT
"
/>
<
add
key
=
"
FINISH_COMMAND
"
value
=
"
FINISH
"
/>
<
add
key
=
"
CONTINUE_COMMAND
"
value
=
"
CONTINUE
"
/>
<
add
key
=
"
MESSAGE_REQUEST
"
value
=
"
R
"
/>
<
add
key
=
"
RECEIVE_COMMAND_FILTER
"
value
=
""
/>
<
add
key
=
"
QueueName
"
value
=
"
WP3_QUEUE
"
/>
<
add
key
=
"
InFileStorageAddress
"
value
=
"
inMessageStorage
"
/>
<
add
key
=
"
OutFileStorageAddress
"
value
=
"
outMessageStorage
"
/>
<
add
key
=
"
MAX_FILE_STORAGE_SIZE
"
value
=
"
1000
"
/>
<
add
key
=
"
DBLoggingReceiver
"
value
=
"
tcp://*:5561
"
/>
<
add
key
=
"
DB_LOGGING
"
valu
e
=
"
false
"
/>
<
add
key
=
"
ClientSettingsProvider.ServiceUri
"
value
=
""
/>
</
appSettings
>
<
log4net
>
<
appender
name
=
"
LogFileAppender
"
type
=
"
log4net.Appender.FileAppender
"
>
<
param
name
=
"
File
"
value
=
"
logs/messenger.log
"
/>
<
param
name
=
"
AppendToFile
"
value
=
"
true
"
/>
<
layout
type
=
"
log4net.Layout.PatternLayout
"
>
<
param
name
=
"
Header
"
value
=
"
[Header]
\
r
\
n
"
/>
<
param
name
=
"
Footer
"
value
=
"
[Footer]
\
r
\
n
"
/>
<
param
name
=
"
ConversionPattern
"
value
=
"
%d [%t] %
-
5p %c %m%n
"
/>
</
layout
>
</
appender
>
<
appender
name
=
"
ConsoleAppender
"
type
=
"
log4net.Appender.ConsoleAppender
"
>
<
layout
type
=
"
log4net.Layout.PatternLayout
"
>
<
param
name
=
"
Header
"
value
=
"
[Header]
\
r
\
n
"
/>
<
param
name
=
"
Footer
"
value
=
"
[Footer]
\
r
\
n
"
/>
<
param
name
=
"
ConversionPattern
"
value
=
"
%d [%t] %
-
5p %c %m%n
"
/>
</
layout
>
</
appender
>
<
root
>
<
level
value
=
"
INFO
"
/>
<
appender
-
ref
ref
=
"
LogFileAppender
"
/>
<
appender
-
ref
ref
=
"
ConsoleAppender
"
/>
<
/
root
>
</
log4net
>
</
configuration
>
D7.1
©
FIRST
consortium
Page
24
of
28
Annex
2
.
Example configuration file for Java component
Annex a
.
Main configuration
<?
xml
version
=
"1.0"
encoding
=
"ISO
-
8859
-
1"
?>
<
messenger
-
definition
>
<
messaging
>
<
ID
>
1
</
ID
>
<
ReceiverNumber
>
1
</
ReceiverNumber
>
<
ProducerNumber
>
1
</
ProducerNumber
>
<
connection
>
<
MessageReceiveAddress
>
tcp://127.0.0.1:5557
</
MessageReceiveAddress
>
<
SendLoadBalancingAddress
>
tcp://127.0.0.1:5558
</
SendLoadBalancingAddress
>
<
FinishReceive
>
tcp://127.0.0.1:5550
</
FinishReceive
>
<
MessageSendAddress
>
tcp
://127.0.0.1:5555
</
MessageSendAddress
>
<
ReceiveLoadBalancingAddress
>
tcp
://127.0.0.1:5556
</
ReceiveLoadBalancingAddress
>
<
FinishPublish
>
tcp
://127.0.0.1:5550
</
FinishPublish
>
<
RECEIVE_COMMAND_FILTER
></
RECEIVE_COMMAND_FILTER
>
<
ACTIVEMQ
>
tcp
://127.0.0.1:61616
</
ACTIVEMQ
>
<
QueueName
>
WP3_QUEUE
</
QueueName
>
</
connection
>
<
MessagingType
>
0
</
MessagingType
>
<
Broker
>
0
</
Broker
>
<
MAX_QUEUE_SIZE
>
10
</
MAX_QUEUE_SIZE
>
<
MIN_QUEUE_SIZE
>
1
</
MIN_QUEUE_SIZE
>
<
BLOCKING_QUEUE
>
FALSE
</
BLOCKING_QUEUE
>
<
IGNORE_QUEUE_OVERFLOW
>
FALSE
</
IGNORE_QUEUE_OVERFLOW
>
<
MESSAGE_REQUEST
>
R
</
MESSAGE_REQUEST
>
<
WAIT_COMMAND
>
WAIT
</
WAIT_COMMAND
>
<
FINISH_COMMAND
>
FINISH
</
FINISH_COMMAND
>
<
CONTINUE_COMMAND
>
CONTINUE
</
CONTINUE_COMMAND
>
<
outFileStorageAddress
>
outMessageStorage
</
outFileStorageAddress
>
<
inFileStorageAddress
>
inMessageStorage
</
inFileStorageAddress
>
<
MAX_FILE_STORAGE_SIZE
>
1000
</
MAX_FILE_STORAGE_SIZE
>
</
messaging
>
</
messenger
-
definition
>
Annex b
.
Logging configuration
<?
xml
version
=
"1.0"
encoding
=
"UTF
-
8"
?>
<!
DOCTYPE
log4j:configuration
SYSTEM
"log4j.dtd"
>
<
log4j:configuration
>
<
appender
name
=
"stdout"
class
=
"org.apache.log4j.ConsoleAppender"
>
<
layout
class
=
"org.apache.log4j.PatternLayout"
>
<
param
name
=
"ConversionPattern"
value
=
"%d{
ABSOLUTE}
%5p %c{1}:%L
-
%m%n"
/>
</
layout
>
</
appender
>
<!
--
log all logs to a separate log file every day
--
>
<
appender
name
=
"orgDevelopersLogFileAppender"
class
=
"org.apache.log4j.DailyRollingFileAppender"
>
<
param
name
=
"file"
value
=
"messenger.log"
/>
<
param
name
=
"datePattern"
value
=
"'.'yyyy
-
MM
-
dd"
/>
<
param
name
=
"append"
value
=
"true"
/>
<
layout
class
=
"org.apache.log4j.PatternLayout"
>
<
param
name
=
"ConversionPattern"
value
=
"%d [%t] %
-
5p %C{6}
(%F:%L)
-
%m%n"
/>
</
layout
>
D7.1
©
FIRST
consortium
Page
25
of
28
</
appender
>
<
root
>
<
priority
value
=
"info"
></
priority
>
<
appender
-
ref
ref
=
"stdout"
/>
<
appender
-
ref
ref
=
"orgDevelopersLogFileAppender"
/>
</
root
>
</
log4j:configuration
>
D7.1
©
FIRST
consortium
Page
26
of
28
Annex
3
.
Configuration of the messaging test scenario
The following chapter contains
a
description of the testbed configuration of messaging
components, used during
the
component development and testing.
The runnable version can
be found on the FIRST machine under the following path: D:
\
ATOS
\
64
\
.
Annex a
.
Configurati
on of components
mimicking WP3
behaviour
(data emitters)
The following listing shows configuration of the testing scenario for WP3 components. The
configuration is the same for both data emitters.
<
configuration
>
<
appSettings
>
<
add
key
=
"
ID
"
value
=
"
1
"
/>
<
add
key
=
"
ReceiverNumber
"
value
=
"
4
"
/>
<
add
key
=
"
ProducerNumber
"
value
=
"
2
"
/>
<
add
key
=
"
MessageSendAddress
"
value
=
"
tcp://127.0.0.1:5555
"
/>
<
add
key
=
"
ReceiveLoadBalancingAdress
"
value
=
"
tcp://127.0.0.1:5561 tcp://127.0.0.1:5562
tcp://127.0.0.1:5563 tcp://127.0.0.1:5564
"
/>
<
add
key
=
"
FinishPublish
"
value
=
"
tcp://127.0.0.1:5550
"
/>
<!
--
0=Pipeline, 1=Request and Reply
--
>
<
add
key
=
"
MessagingType
"
value
=
"
0
"
/>
<
add
key
=
"
MAX_QUEUE_SIZE
"
value
=
"
5
"
/>
<
add
key
=
"
MIN_QUEUE_SIZE
"
value
=
"
1
"
/>
<
add
key
=
"
IGNORE_QUEUE_OVERFLOW
"
value
=
"
0
"
/>
<
add
key
=
"
Broker
"
value
=
"
0
"
/>
<
add
key
=
"
ACTIVEMQ
"
value
=
"
tcp://localhost:61616
"
/>
<
add
key
=
"
WAIT_COMMAND
"
value
=
"
WAIT
"
/>
<
add
key
=
"
FINISH_COMMAND
"
value
=
"
FINISH
"
/>
<
add
key
=
"
CONTINUE_COMMAND
"
value
=
"
CONTINUE
"
/>
<
add
key
=
"
MESSAGE_REQUEST
"
value
=
"
R
"
/>
<
add
key
=
"
RECEIVE_COMMAND_FILTER
"
value
=
""
/>
<
add
key
=
"
QueueName
"
value
=
"
WP3_QUEUE
"
/>
<
add
key
=
"
InFileStorageAddress
"
value
=
"
inMessageStorage
"
/>
<
add
key
=
"
OutFileStorageAddress
"
value
=
"
outMessageStorage
"
/>
<
add
key
=
"
MAX_FILE_STORAGE_SIZE
"
value
=
"
1000
"
/>
<
add
key
=
"
DBLoggingReceiver
"
value
=
"
tcp://*:5561
"
/>
<
add
key
=
"
DB_LOGGING
"
value
=
"
false
"
/>
<
add
key
=
"
ClientSettingsProvider.ServiceUri
"
value
=
""
/>
</
appSettings
>
<
/
configuration
>
Annex b
.
Configuration
for mimicking
WP4 behaviour
(data
receivers and emitters)
In the testbed scenario, there are four WP4 components. The following configuration
applies to
the first component:
<
messenger
-
definition
>
<
messaging
>
<
ID
>
1
</
ID
>
<
ReceiverNumber
>
2
</
ReceiverNumber
>
<
ProducerNumber
>
4
</
ProducerNumber
>
<
connection
>
D7.1
©
FIRST
consortium
Page
27
of
28
<
MessageReceiveAddress
>
tcp
://127.0.0.1:5555
tcp
://127.0.0.1:5556
</
MessageReceiveAddress
>
<
SendLoadBalancingAddress
>
tcp
://127.0.0.1:5561
</
SendLoadBalancingAddress
>
<
FinishReceive
>
tcp
://127.0.0.1:5550
</
FinishReceive
>
<
MessageSendAddress
>
tcp
://127.0.0.1:5557
</
MessageSendAddress
>
<
ReceiveLoadBalancingAddress
>
tcp
://127.0.0.1:5565
tcp
://127.0.0.1:5566
</
ReceiveLoadBalancingAddress
>
<
RECEIVE_COMMAND_FILTER
></
RECEIVE_COMMAND_FILTER
>
<
ACTIVEMQ
>
tcp
://127.0.0.1:61616
</
ACTIVEMQ
>
<
QueueName
>
WP3_QUEUE
</
QueueName
>
</
connection
>
<
MessagingType
>
0
</
MessagingType
>
<
Broker
>
0
</
Broker
>
<
MAX_QUEUE_SIZE
>
10
</
MAX_QUEUE_SIZE
>
<
MIN_QUEUE_SIZE
>
1
</
MIN_QUEUE_SIZE
>
<
BLOCKING_QUEUE
>
FALSE
</
BLOCKING_QUEUE
>
<
IGNORE_QUEUE_OVERFLOW
>
FALSE
</
IGNORE_QUEUE_OVERFLOW
>
<
MESSAGE_REQUEST
>
R
</
MESSAGE_REQUEST
>
<
WAIT_COMMAND
>
WAIT
</
WAIT_COMMAND
>
<
FINISH_COMMAND
>
FINISH
</
FINISH_COMMAND
>
<
CONTINUE_COMMAND
>
CONTINUE
</
CONTINUE_COMMAND
>
<
outFileStorageAddress
>
outMessageStorage
</
outFileStorageAddress
>
<
inFileStorageAddress
>
inMessageStorage
</
inFileStorageAddress
>
<
MAX_FILE_STORAGE_SIZE
>
1000
</
MAX_FILE_STORAGE_SIZE
>
</
messaging
>
</
messenger
-
definition
>
Second c
omponent
(
showing
only differing lines
)
:
<
SendLoadBalancingAddress
>
tcp
://127.0.0.1:5562
</
SendLoadBalancingAddress
>
<
MessageSendAddress
>
tcp
://127.0.0.1:5558
</
MessageSendAddress
>
Third component (
showing
only d
iffering lines
):
<
SendLoadBalancingAddress
>
tcp
://127.0.0.1:5563
</
SendLoadBalancingAddress
>
<
MessageSendAddress
>
tcp
://127.0.0.1:5559
</
MessageSendAddress
>
Fourth comoponent (
showing
only differing lines
):
<
SendLoadBalancingAddress
>
tcp
://127.0.0.1:5564
</
SendLoadBalancingAddress
>
<
MessageSendAddress
>
tcp
://127.0.0.1:5560
<
/
MessageSendAddress
>
D7.1
©
FIRST
consortium
Page
28
of
28
Annex c
.
Configuration
for
mimicking WP6 behaviour (data
receivers only)
The following listing shows the configuration
of the first of two WP6 components
.
<
configuration
>
<
appSettings
>
<
add
key
=
"
ID
"
value
=
"
1
"
/>
<
add
key
=
"
ReceiverNumber
"
value
=
"
2
"
/>
<
add
key
=
"
ProducerNumber
"
value
=
"
4
"
/>
<
add
key
=
"
MessageReceiveAddress
"
value
=
"
tcp://127.0.0.1:5557 tcp://127.0.0.1:5558
tcp://127.0.0.1:5559 tcp://127.0.0.1:5560
"
/>
<
add
key
=
"
SendLoadBalancingAddress
"
value
=
"
tcp://127.0.0.1:5565
"
/>
<
add
key
=
"
FinishReceive
"
value
=
"
tcp://127.0.0.1:5550
"
/>
<!
--
0=Pipeline, 1=Request and Reply
--
>
<
add
key
=
"
MessagingType
"
value
=
"
0
"
/>
<
add
key
=
"
MAX_QUEUE_SIZE
"
value
=
"
5
"
/>
<
add
key
=
"
MIN_QUEUE_SIZE
"
value
=
"
1
"
/
>
<
add
key
=
"
IGNORE_QUEUE_OVERFLOW
"
value
=
"
0
"
/>
<
add
key
=
"
Broker
"
value
=
"
0
"
/>
<
add
key
=
"
ACTIVEMQ
"
value
=
"
tcp://localhost:61616
"
/>
<
add
key
=
"
WAIT_COMMAND
"
value
=
"
WAIT
"
/>
<
add
key
=
"
FINISH_COMMAND
"
value
=
"
FINISH
"
/>
<
add
key
=
"
CONTINUE_COMMAND
"
value
=
"
CONTINUE
"
/>
<
add
key
=
"
MESSAGE_REQUEST
"
value
=
"
R
"
/>
<
add
key
=
"
RECEIVE_COMMAND_FILTER
"
value
=
""
/>
<
add
key
=
"
QueueName
"
value
=
"
WP3_QUEUE
"
/>
<
add
key
=
"
InFileStorageAddress
"
value
=
"
inMessageStorage
"
/>
<
add
key
=
"
OutFileStorageAddress
"
value
=
"
outMessageStorage
"
/>
<
add
key
=
"
MAX_FILE_STORAGE_SIZE
"
value
=
"
1000
"
/>
<
add
key
=
"
DBLoggingReceiver
"
value
=
"
tcp://*:5561
"
/>
<
add
key
=
"
DB_LOGGING
"
value
=
"
false
"
/>
<
add
key
=
"
ClientSettingsProvider.ServiceUri
"
value
=
""
/>
</
appSettings
>
</
configuration
>
The
configuration of the second components
is the same except for the following line:
<
add
key
=
"
SendLoadBalancingAddress
"
value
=
"
tcp://127.0.0.1:5566
"
/>
Enter the password to open this PDF file:
File name:
-
File size:
-
Title:
-
Author:
-
Subject:
-
Keywords:
-
Creation Date:
-
Modification Date:
-
Creator:
-
PDF Producer:
-
PDF Version:
-
Page Count:
-
Preparing document for printing…
0%
Comments 0
Log in to post a comment