Middleware Progress report and plans - JINR

fortnecessityusefulDéveloppement de logiciels

14 déc. 2013 (il y a 3 années et 8 mois)

178 vue(s)

Wojciech
Sliwinski

Wojciech.Sliwinski@cern.ch

Beams

Department, Controls Group

CERN

Outline


Defining

Middleware



Types

of Middleware



CERN
Acc
. Controls
Messaging

Infrastructure



Middleware
Review

for CERN
Acc
. Controls



Conclusions


2

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Outline


Defining

Middleware



Types

of Middleware



CERN
Acc
. Controls
Messaging

Infrastructure



Middleware
Review

for CERN
Acc
. Controls



Conclusions


3

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Why

Middleware?



How do we build/connect
applications over a
network?



How

do we
facilitate

Distributed
C
omputing

(2
-
tier, …,
n
-
tier
)?



How

to
support
the

heterogeneous
environments
?

26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

4

What is Middleware?


Software
,

which allows an application
to interoperate
with other
software, without requiring the user
to understand
and
to
code
the low
-
level operations required to achieve interoperability


S
oftware layer between
OS

and the applications


Hides

complexity

&
heterogenity

of
distributed

system


Handles

issues

related

to

OS,
Network Protocols

&

hardware
platforms


5

Application

Operating

System1

Communication

system

MW

Application

Operating

System2

MW

Application

Operating

SystemN

MW

Standard API

Standard API

Specific

API

Specific

API

Standard API

Specific

API

26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

Introduction to Middleware


Middleware provides support for:


Naming, Location, Service discovery, Replication


Protocol handling, Communication faults,
QoS


Synchronisation, Concurrency,
Failover
,
Scalability


Access control, Authentication



Middleware dimensions:


Request
-
Reply


vs.

Asynchronous Messaging


Language
-
specific

vs.

Language
-
independent


Proprietary


vs.

Standards
-
based


Small
-
scale


vs.

Large
-
scale


Tightly
-
coupled


vs.

Loosely
-
coupled components

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

6

Middleware is everywhere.

You might think you can leave it but it

just finds you again

:
-
)

26th October 2011

Outline


Defining

Middleware



Types

of Middleware



CERN
Acc
. Controls
Messaging

Infrastructure



Middleware
Review

for CERN
Acc
. Controls



Conclusions


7

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Middleware


large

domain


which

to
choose
?

8

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011








direct
p2p

broker

req
/rep

pub/sub

rpc
/orb

message

LAN

WAN/

Internet

Network

Embedded

Types

of Middleware


Remote

Procedure

Call

(RPC)



Object
-
Oriented

Middleware (OOM)



Message
-
Oriented

Middleware (MOM)



Event
-
Based

Middleware
a.k.a

Publish
-
Subscribe

26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

9

Remote

Procedure

Call

-

RPC


Masks remote function calls as being local


Client
-
Server

model
with

Request
-
Reply
paradigm


S
ynchronous

c
ommunication


Marshalling of function parameters and return value


Provides
L
ocation

& P
latform

transparency


Technologies:


DCE
-
RPC


XML
-
RPC


26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

10

Client App.

IDL
Stub

Network

Server

App.

IDL Skeleton

Network

Req
/
Reply

RPC
-

Disadvantages


Synchronous
R
equest
-
R
eply

interaction


T
ight

coupling between client and server


C
lient

may block for a long time if server loaded


M
ulti
-
threaded programming at client


S
low/failed clients may delay servers when replying


M
ulti
-
thre
aded

programming

a
t servers



RPC paradigm is not object
-
oriented


I
nvoke

functions on servers as opposed to methods on objects

26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

11

Object
-
Oriented

Middleware
-

OOM


Client
-
Server

model
with

Request
-
Reply
paradigm


S
ynchronous

c
ommunication


Marshalling of function parameters and return value


Provides
L
ocation

& P
latform

transparency


Objects

can

be
local

or

remote


R
emote objects
accesible

via
proxy objects


Technologies:


OMG
CORBA


MS
DCOM


Java
RMI


SOAP

26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

12

Client App.

IDL
Stub

IIOP

Client

ORB

Server

App.

IDL Skeleton

IIOP

Server ORB

POA Adapter

Req
/
Reply

Object
-
Oriented

MW
-

Disadvantages


Synchronous
R
equest
-
R
eply

interaction


T
ight

coupling between client and server


C
lient

may block for a long time if server loaded


M
ulti
-
threaded programming at client


S
low/failed clients may delay servers when replying


M
ulti
-
thre
aded

programming

a
t servers



Distributed garbage collection


Releasing memory for unused remote objects


Error
-
prone

and
difficult

to
debug



S
tatic

mappings

and heavy
-
weight

development


Difficult

to
extend

an
operational

system

26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

13

Message
-
Oriented

Middleware
-

MOM


Data
exchange

via
Messages

&
Message

Queues


Asynchronous
c
ommunication

(
Point
-
to
-
point



One
-
to
-
One
)


Client

and Server
are

loosely

coupled


Message

Servers

may

do
filtering
,
transforming
,
logging



Technologies: IBM

WebSphereMQ
,
TIBCO Rendez
-
vous
,
JMS
, SOAP


26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

14

Client App.

local message

queues

Server App.

local message

queues

message

queues

Network

Network

Network

Message
Servers

Message
-
Oriented

MW
-

Disadvantages


Poor programming abstraction (but has evolved)


Rather low
-
level


Request
-
R
eply more difficult to achieve, but can be done



Message formats originally unknown to middleware


No type checking (
solved

in

JMS)



Queue abstraction only gives
O
ne
-
to
-
O
ne communication


Limits scalability (
solved

in

JMS

with

Publish
-
Subscribe
)


26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

15

What we lack, so far


General interaction patterns


W
e have
R
equest
-
R
eply

and
O
ne
-
to
-
O
ne


Publish
-
Subscribe
? O
ne
-
to
-
M
any?
M
any
-
to
-
M
any?


D
ynamic

joining and leaving?



Location transparency


A
nonymity
of communicating entitie
s

26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

16

Event
-
Based

Middleware
a.k.a

Publish
-
Subscribe


Publishers (advertise and) publish
E
vents (
M
essages
)


Subscribers express interest in
E
vents with
S
ubscriptions


Event Service notifies interested
S
ubscribers

of published
E
vents


Events can have arbitrary content (typed)


Technologies (
Brokers
): JMS (
ActiveMQ
,
RabbitMQ
), AMQP (…)


26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

17

Publisher1

Event

Service

(Broker)

PublisherN



Publisher2

topics

publish

Subscriber1

SubscriberN



Subscriber2

subscribe

notify

Properties of Publish
-
Subscribe


Asynchronous communication


Publishers and
S
ubscribers are loosely coupled



Many
-
to
-
M
any interaction between
P
ubs and
S
ubs


Scalable scheme for large
-
scale systems


Publishers do not need to know
S
ubscribers, and vice
-
versa


Dynamic join and leave of
P
ubs

&

S
ubs (
thanks

to B
rokers)



Topic based
Publish
-
Subscribe

very expressive


Filtered information delivered only to interested parties

26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

18

Outline


Defining

Middleware



Types

of Middleware



CERN
Acc
. Controls
Messaging

Infrastructure



CERN
Acc
. Controls Middleware
Review



Conclusions


19

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

CERN Controls Middleware


in

all

SW
tiers


20

CMW
client

(C++/Java)

GUIs
,
LabView
, RADE

CMW
client

(Java
servers
)

Logging
, LSA,
InCA
, SIS

CMW
server

(C++/Java)

Proxy, DIP,
AlarmMon
, AQ

CMW
server

(C++)

FESA, FGC, GM

CMW
server

(C++)

PVSS (
Cryo
,
Vacuum
)

JMS
client

(Java)

GUIs

JMS
client

(Java)

Servers
:
Logging
,
InCA
, SIS

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

JMS for Accelerator Controls


J
ava
M
essaging
S
ervice


Purpose


Reliable transport of data between Java Processes


Controls


Alarm System


Data Rendering Services


(Control) GUIs


Logging Services


Beam Security System(s)



No JMS
-

No Beam !

26th October 2011

21

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

JMS for Accelerator Controls

Devices / Hardware

JMS

C/C++ Process


JAVA Process

JAPC

CMW

JMS

Front End


GUI Process

JAPC

CMW

JMS

CMW

CMW

~50 Processes

~
100

Processes

~46K Subscriptions

280 Connections

40
00 servers @

1
7
00 machines

26th October 2011

22

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

Examples

of JMS ”
aware
” systems

26th October 2011

23

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

History

of JMS
in

Accelerator

Controls


Usage of
SonicMQ

JMS for LHC Alarm System since 2001



ActiveMQ

introduced in 2005



Other project started to use JMS service



Today vital part of Beam Operations

26th October 2011

24

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

Technical Aspects


Broker Configuration


Non
-
persistent messages on topics


No subscription recovery policy


No durable subscribers


No exclusive consumers


No
replyTo

mechanism


Broker is persistent



Clients


Pure JAVA
-

Wireformat

Protocol


26th October 2011

25

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

General JMS Deployment Situation


SonicMQ

: 12 Brokers


TIM, LASER, DIAMON


Always as two node cluster



ActiveMQ

: 16 Brokers


One main
default Broker Service


Otherwise
project dedicated

Brokers



All Brokers on Technical Network


26th October 2011

26

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

Overall Broker Service Jan 2011


26th October 2011

27

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

JMS Service
Data Handling


Central JMS Service (276 days in 2010)


58 Million
msg

/ day


120 TB IN


650 TB OUT



Additional handling of


Heavy Connection and

Subscription handling


Bursts loads


Connected subscribers grew

from 2K to 12K in 6 months


=> 440 GB /day

=> 2.5 TB /day

YES,
ActiveMQ


can!

26th October 2011

28

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

Data Handling during LHC startup 2010

Total Data Out

Increase in data demand

2.57 TB/day

1TB/day

LHC startup

26th October 2011

29

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

JMS
Service Availability 2010*


Total hours : 6624



ActiveMQ
:

= 8 minutes downtime (due to kernel update)



SoniqMQ

:

= 8 hours downtime

* November 2010

26th October 2011

30

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

Monitoring and Troubleshooting Tools


DIAMON & Lemon



JConsole

/
JVisualVM



Dump of broker information to
SQLite

DB



Traffic Monitoring Tool



Various Message Listeners


26th October 2011

31

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

Easy Identification of Bad Producers

Fixed

Problematic Producer

26th October 2011

32

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

JMS Service


Lessons

Learnt



ActiveMQ

was
a

good choice


Running reliably


Performs well


Provides many metrics for diagnostic and monitoring


Good integration in BE
-
CO monitoring tools via JMX



Projects
are not

always

aware

of sending

so much data


Service Level Agreements with projects required


More analyzing tools for end users required



26th October 2011

33

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

JMS Service


Lessons

Learnt



Monitoring is a
vital part

of operations


Message reception speed test


Memory Percent usage


JMX attributes (#topics, #consumers, #subscriptions)



Broker Configuration is not always
obvious



26th October 2011

34

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

JMS Service


Lessons

Learnt



Clustering


Good for
failover

and
load balancing


Additional
complexity


Updating clients for new cluster members


Automatic
rebalancing of clients



Memory is important


Tuning

of Java
GC

options


26th October 2011

35

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

Outline


Defining

Middleware



Types

of Middleware



CERN
Acc
. Controls
Messaging

Infrastructure



CERN
Acc
. Controls Middleware
Review



Conclusions


36

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Hardware Devices

GUI operator consoles

Front ends

“virtual devices”

CERN Controls Middleware


CMW

37

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Controls Middleware

Embedded Middleware

CMW
Mandate

&
Scope


Standard

set of MW solutions

&
libraries


Centrally

managed services


Track & optimize
runtime parameters



Scope:
CERN Accelerator Complex


Operational
24*7*365


Must be
Reliable & High Quality


~80
’000 HW devices,
~4

000

servers


~1700
machines
, ~2’000’000 IO
points



Huge

infrastructure
!


38

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

CMW Data Flow

39

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Linux/Windows

CERN Controls Middleware

40

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

C++/Java process

Controls Middleware API

TCP/IP

CORBA

Controls Middleware

RT Linux/
LynxOS

C++ process

Controls Middleware API

CORBA

Problems:

-

C++ and Java implementations differ




-

Heavy in memory usage





-

Complex error prone API






-

No direct support for pub/sub





-

Blocking issues (
JacORB
)





-

Shrinking community






-

Lack of new releases and bug fixes









Long LHC shutdown at the end of 2012

A unique possibility!

Instead of patching let’s provide a new one!

CERN Middleware Requirements

41

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Desirable

Mandatory

Fundamental

Lightweight

Active community

Friendly API, documentation

Request/reply & pub/sub patterns

Open source, redistributable license

QoS

Stability, Maturity & Longevity

Performance & Scalability

Asynchronous

C++/Java

Linux/Windows

Over TCP/IP LAN

How did we evaluate

> our criteria

42

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

CRITERIA






QoS

resources,

binary size,

memory

performance

Communications

patterns

API, look & feel,

documentation

Community,

maturity

Appearance


Creators


specification


documentation


Users


forums


bug reports


Internet

Simple usage


Download


licensing


Compile


Linux


LynxOS


Run examples


Testing


Communication
patterns


Performance


QoS


Exceptional
situations

Surprising how many great new products!

43

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

BSD

sockets

RPC

CORBA

???

2011

1969

TODAY

1974

1998

1980

1981

1983

1991

2011

Evaluated middleware products

44

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Ice

Thrift

omniORB

YAMI

OpenSpliceDDS

OpenAMQ

CoreDX

RTI DDS

ZeroMQ

QPid

MQtt

RSMB

JacORB

Mosquito

All
opinions

are based only on
our knowledge
and
evaluation
. Each of the
products, depending on the requirements, may constitute a good solution.

RabbitMQ

CORBA
(
omniORB
,
JacORB
)

45

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Object
-
oriented communication platform standardized by OMG. It is
over 20 years old, and the standard is well established but...



Complex, hard to learn and use





Differences between implementations




Big memory footprint







Shrinking community






Seldom updates and bug fixes





Object
-
oriented middleware by
ZeroC
, people who left CORBA to fix
its problems. Conceptually similar to CORBA but...



Better design and implementation





Modern easier to use API and IDL mapping




Concise C++ and Java implementation





Support for versioning







Active community







Ice

46

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Thrift

47

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Middleware

by
Facebook

(moved to Apache)



Lightweight, free of unnecessary dependencies





Modern, clean API








Active community









Ongoing development, still incomplete





No documentation, empty tutorial





Message
-
oriented middleware developed
by one of our former
colleagues. Used
by some CERN Controls services.



Lightweight, free of unnecessary dependencies






Well designed, modern API







Asynchronous
req
/rep and pub/sub patterns





Dynamic type specification (paid in performance)





Small community








YAMI4

48

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Middleware where messages are distributed by a broker. A few
independent implementations of the broker and clients.



Big community










Designed to support pub/sub







Protocol standard issues








Broker


single point of failure







Broker


additional hop, slower communication





Broker


additional, non compliant monitoring tools




AMQP


OpenAMQ
,
Qpid
,
RabbitMQ

49

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Data
-
oriented middleware with p2p communication. Standard
defined by OMG, with a few compatible implementations.



Big industrial and military community




Possible compatibility with CORBA




Designed to support pub/sub





Req
/rep possible but needs 2 channels




Steep learning curve






Complex API








DDS


RTI,
CoreDX
,
OpenSplice

50

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Message
-
oriented middleware by
iMatix

(they dropped
OpenAMQ
.)
API resembles BSD sockets but it is so much more.



Simple yet powerful API








Support for in
-
proc, inter
-
proc, TCP, PGM






Lightweight, free of unnecessary dependencies







Many communication patterns







May be used as a concurrency framework





Active community








ZeroMQ

51

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Performance requirements reviewed

52

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Instead of estimating


We asked our users what they will need


Gathered statistics from the current system



Most demanding users identified


Definition of a few performance tests



C++ server
with client in:

Performance tests, reliable
R
eq
uest
-
Reply

53

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

4000
msg
/sec

Payload = 4B

5
msg
/sec

Payload = 10MB

Many small
messages

A few big
messages

Performance/Scalability tests, reliable
P
ub
-
S
ub

54

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

400
msg

x 8 B

10 clients, <50ms

30
msg

x 8 B

10 clients, <20ms

Throughput

Latency

Products comparison (according to the criteria)

55

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Sync,
async

&
msg

patterns

QoS

Dependencies

& memory f
-
p

Performance

Look & feel,

API, docs

Community &
maturity

Score

ZeroMQ













6

Ice













5

YAMI4













4

RTI













3

Qpid













3

CORBA













2

Thrift













2

Middleware
Review

Summary


Several good middleware
solutions available.


The choice is dictated by the
most critical requirements
for any given application. Not easy


QoS

policies and
performance matter, but also ease of use, community, …

56


Concerning CERN Controls Middleware…


P
rototype

with the most

promising candidates:




Deploy the new middleware before the long
accelerator shutdown

at the end of 2012


ZeroMQ

Ice

YAMI

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

26th October 2011

Conclusions


Middleware is an
important abstraction
for building


the

distributed systems



Synchronous

vs.
asynchronous

communication



One
-
to
-
one

vs
.
many
-
to
-
many

communication



Better

performance

with

direct

communication



Better

scalability

and less
coupling

with

Brokers



System
requirements



Messaging

patterns

to
apply

26th October 2011

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

57