Felix Ehm CERN BE-CO

californiamandrillΛογισμικό & κατασκευή λογ/κού

13 Δεκ 2013 (πριν από 3 χρόνια και 7 μήνες)

120 εμφανίσεις

Felix
Ehm

CERN BE
-
CO


Content


Introduction



JMS in the Controls System



Deployment and Operation



Conclusion

Introduction to JMS


The API


J
ava
M
essaging
S
ervice


Messaging API defined in 2002


Decouples

Source and Destination


Topic / Queue Concepts for named Destinations






Introduction to JMS


A Broker


Technology originated from the financial
sector


“Few” sources and large set of readers



A
Broker

is the central entity


Accepts messages from
1..*
publisher

and

re
-
distributes to
0..*

subscribers


Provides Quality of Service for Messages


Persistency, Transactions, Buffering, Expiration Policies,
Slow Client Handling


Design to distribute data
reliably

and
scalable


Load balancing / fail
-
over / embedded

deployment scenarios possible




Broker

Introduction to JMS
-

A Broker


Many broker implementations available


Proprietary

IBM
WebsphereMQ
,
SonicMQ
, TIBCO,
RedHat

MRG, …


OpenSource

Apache
ActiveMQ
, Apache Apollo,
RabbitMQ
,
HornetMQ
, …



Broker may be implemented in another language
than Java (e.g.
Qpid

&
RabbitMQ
)



Communication protocol is non
-
standard


Brokers of different vendors are
not interoperable


AMQP

&
STOMP

address this issue

Introduction to JMS
-

Request/Reply


“Direct communication” via Broker using

Request


Reply
mechanism









Due to intermediate Broker two hops are required

Requestor

Replier

Broker

Request

MSG

Reply

MSG

Request Queue

Temporary Reply

Queue

creates

sends

sends

Introduction to JMS
-

Selectors


Filter messages from one Destination using

Messages Selectors


Each message can be enriched

with Attributes


Subscriber receives filtered

messages via defined selector

Broker

Subscriber

s
elector:

BUILDING ≠ A

HEADERS

CYCLE=A

TIME=9845

BODY

JMS Message

BUILDING=A

BUILDING=B

BUILDING=C

Topic/
Queue

Introduction to JMS
-

Persistency


JMS defines
persistent

and
non
-
persistent

messages


Broker stores message to

disk
before acknowledging

to sender



Avoids loss of data when broker crashes


Drawbacks


Higher disk load


Slower message processing


Sender

1. send

3.
ack

2
. store

CERN’s Accelerator Complex


CERN’s Accelerator Complex


JMS in the Controls System


Purpose


Reliable transport of data between (Java) Processes


High
l
evel
c
ontrols applications + middle tier servers


Alarm System


Data Rendering Services


(Control) GUIs


Logging Services


Beam Security System(s)


Access Systems, ….



Today
vital part of Beam
Operations


Product choice:


No JMS
-

No Beam !

JMS in the Controls System
-

History


History


SonicMQ

since 2001 for CERN’s Alarm

and CO Monitoring system


OpenSource

ActiveMQ

since 2005


Middle tier servers update GUIs


Middle tier to Middle tier



Phased out
in 2013

JMS for Accelerator Controls


Early days only Java clients via JAPC


japc
-
ext
-
remote contains JMS communication



Today multi
-
language clients


JAPC clients remain


Java using
ActiveMQ

client libraries directly


C++/Python using STOMP



29 production brokers


Single or broker cluster




Large spectrum of usage patterns


from few topics to 40K topics


from few subscribers to >23 K subscribers


from couple
msg

/ minute to 10K
msg

/ sec


from few bytes/
msg

to 3
Mbyte
/
msg



Deployment and Operation

Deployment and Operation

Some numbers :


300 Applications


4’400 Connections


40’000 Subscriptions


85’000 Topics


8 Million
msg
/h IN,

3.5M
msg
/h OUT


Archived Uptime

in 2012: 99.998%

Not all data which
is produced is also
consumed

Deployment and Operation


Distribute Load


Interconnect two Brokers


Consumers/Producer can

choose any Broker


Sharing of work



Smooth Upgrades possible


C
lient library version != Broker version
OK



Drawbacks


Consumer might not get all messages in case of

network error


Routing overhead

The
STOMProtocol


The
STOMProtocol


S
imple
T
ext
O
riented
M
essage
P
rotocol


Defines messaging API + wire format


Many open source clients libraries available


Python, C/C++, Java, Perl, Ruby, PHP,
Erlang
, …


Targets the messaging interoperability among
language, platforms and brokers


Supported by
many brokers


Very easy to use

Example to send STOMP in Python

mysocket

=
socket.socket
(
socket.AF_INET,socket.SOCK_STREAM
)

mysocket.connect
( ("
jms
-
co
-
dev
", 61680) )


text = '''SEND
\
n
\

destination
:/queue/CERN.DEPLOYMENT
\
n
\

HOST:%s
\
n
\

INSTALLPATH:%s
\
n
\

PRODUCT:%s
\
n
\

TIME
:%s
\
n
\

persistent:false
\
n
\
n
\
x00
''‘


mysocket.sendall
(text
)

mysocket.sendall
("DISCONNECT
\
x00")

mysocket.close
()

The
AMQProtocol


The
AMQProtocol


A
dvanced
M
essage
Q
ueue
P
rotocol


Defines messaging API + wire format


Next evolution after JMS


Consortium of
RedHat
, Credit Suisse, Microsoft,
JPMorgen
, Barclays,
VMWare
, Cisco, …


Released as Version 1.0


More and more brokers implement it



Direction for the future ?

Conclusion


JMS is
vital part

of
Accelerator Controls System



Performs very well for publish
-

subscribe scenarios



ActiveMQ

was
a

good choice


Running reliably


Low maintenance and flexible deployments


Provides many metrics for diagnostic and monitoring



Multi
-
language
applications can use service



Central (redundant) service






Result Example


Messages received/sec over time

time

AVG Messages / sec

Result Example


Histogram of delayed message distribution

Delay time [
msec
]

# Messages

Result Example


Number of “late” (>300ms) messages for test
duration

time

Delayed Messages

Message Delay Summary


Measuring delayed Message


~ 99.32% of messages delivered < 300ms


Rest < 1 sec



Doubling numbers of messages

(~9500
msg
/sec)


100% between 5 and 10 seconds


Further studies for improvement needed

40% less CPU load thanks to optimization

Result Example


CPU load


time

% CPU load

Consumer
connects

Consumer
disconnects

Broker Memory Consumption


Initial Memory is proportional to


Connections


Subscriptions



0
50
100
150
200
250
300
350
400
0
5000
10000
20000
30000
40000
Heap Memory Usage
Live Threads
0
100
200
300
400
500
600
700
800
0
10000
20000
40000
60000
80000
Heap Memory Usage
Live Threads
1000 Subscription / Connection

500 Subscriptions / Connection