Open Source EDA

sixmileugliestInternet και Εφαρμογές Web

24 Ιουν 2012 (πριν από 5 χρόνια και 2 μήνες)

353 εμφανίσεις

Open Source
EDA

Edward Yavno

Sr. Associate Consultant at Lab49

Twitter: @
edyavno



Open Source EDA, Edward Yavno, Lab49

1

Open Source
EDA

EDA = Event Driven Architecture

“Open Source EDA” Inspired by:




High quality Open Source products that




can be used as building blocks for EDA




img
: http://www.flickr.com/photos/hansandcarolyn





“Open Source SOA” by Jeff Davis (Manning)

Open Source EDA, Edward Yavno, Lab49

2

Introducing Open Source EDA Java Stack


ActiveMQ
as a message broker


Ehcache / Terracota
as distributed
cache /
in
-
memory
data store


Esper as a CEP Engine


JGroups


reliable
multicast


QuickFix/J


FIX
engine

Open Source EDA, Edward Yavno, Lab49

3

Demo Time

Two Slightly Different Demos

EDA Demo
#1

Simple Foreign Exchange (FX)
Trading
System (EMS)








Open Source EDA, Edward Yavno, Lab49

5

EDA Demo
#2

Real Time Web App that
aggregates Twitter
mentions of programming languages per state

Open Source EDA, Edward Yavno, Lab49

6

What is EDA?

Event: anchored in time, past action or state change.

Event
-
driven architecture (EDA) is a software architecture pattern promoting the
production, detection, consumption of, and reaction to events (Wikipedia).




Gregor

Hohpe

(
http://www.eaipatterns.com
):


Broadcast
communications: broadcast events to any interested party


Timeliness
: publish events as they occur


Asynchrony


More
fine
-
grained events


Complex
Event Processing: The system understands and monitors the
relationships between
events

Open Source EDA, Edward Yavno, Lab49

7

What is EDA? (cont’d)

Brenda
Michelson (
http://www.elementallinks.com
):


Extremely
loosely coupled, and highly
distributed … best
used for
asynchronous flows of work and information


3
styles of event processing are often used together in a mature EDA system:
simple, stream, and
complex


Event
Flow Layers

1.
Event
Generators

2.
Event
Channel

3.
Event
Processing

4.
Downstream
Event
-
Driven Activity


Open Source EDA, Edward Yavno, Lab49

8

Apache ActiveMQ


Message Broker written in Java with JMS support (and much more!)


Message Broker


way to decouple
the destination of a message from the
sender and maintain central control over the flow of
messages
(
eaipatterns.com
)


Event Channel Layer in
EDA

Features:


Supports
pluggable transport protocols such as in
-
VM, TCP, SSL
, Java
NIO,
UDP, multicast, JGroups and JXTA
transports


Many clustering options: failover, auto
-
discovery, network of brokers,
etc


Supports fast configurable persistence stores (for queues, durable topics
)


Interesting: support for REST API, Ajax,
recently
for HTML5 WebSockets


Integration with web
service
stacks (CXF, Axis2) for reliable messaging


Different topologies:
in VM, Client
-
server, Embedded Broker, Peer
to
Peer

Open Source EDA, Edward Yavno, Lab49

9

API: JMS

JMS
30 second review
:

Queue,
Topic (pub/sub),
Durable
Topic

Publisher:



queueSender.send
(message
);



topicPublisher.publish
(message
);

Polling
Consumer:



Message m =
receiver.receive
();

Event
-
Driven
Consumer
:



public interface
MessageListener

{






void
onMessage
(Message
msg
);




}

Open Source EDA, Edward Yavno, Lab49

10

Ehcache


Distributed Cache … or Distributed In Memory Key
-
Value Data
Store


Can be Event Generator, Event Channel, Downstream
Event
-
Driven
Activity Layer in EDA

Features:


Fast and Light weight; memory and disk stores, supports
sync/
async

replication, different replication mechanisms
;
expiry
policies; LRU, LFU and FIFO cache eviction policies


Replication via Terracota, RMI,
JGroups,
JMS


Support for Cache Event notifications
-

event listeners!!!




Open Source EDA, Edward Yavno, Lab49

11

Ehcache API

Map
-
like API for CRUD:

Cache
myCache

=
CacheManager.getInstance
().
getCache
(
"
myCache
"
);

myCache.put
(
new

Element(
"
myKey
"
,
"
myValue
"
) );

myCache.put
(
new

Element(
"
myKey
"
,
"
myValueAnother
"
) );

//
updates the entry for "
myKey
"

Element
element

=
myCache.get
(
"
myKey
"
);
Serializable

value =
element.getValue
();

boolean

foundAndRemoved

=
myCache.remove
(
"
myKey
"
);

Cache Event Listeners:

interface

CacheEventListener



void

notifyElementRemoved
(
final

Ehcache cache,
final

Element element)
throws

CacheException
;

void

notifyElementPut
(
final

Ehcache cache,
final

Element element)
throws

CacheException
;

void

notifyElementUpdated
(
final

Ehcache cache,
final

Element element)
throws

CacheException;

void

notifyElementExpired
(
final

Ehcache cache,
final

Element element
);

Open Source EDA, Edward Yavno, Lab49

12

3

½ Minute
Intro to CEP

Database upside down!

Twitter
Marriage Proposal Example
:





@guy1024
: @gal1024 Will you
marry
me???!!!





August 16, 2010 6:30:00 PM from New York, NY

@gal1024
: @guy1024 Yes, I will!!!




August 16, 2010 6:34:00 PM from New York, NY




Questions (Complex Events):


Did
she respond within 5 minutes?


How many other successful proposals in New York within last 10
mins

at any
point in time?


When did New York
exceed California
by more than 15% in number of
successful proposals on Twitter within any 10 min interval?


Open Source EDA, Edward Yavno, Lab49

13

Esper


CEP & ESP Engine


CEP
: processing
events
to discover
complex patterns among multiple streams of
event
data; ESP: processing
multiple streams of event data
to
identify
the meaningful
events within those
streams


Event Processing Layer in
EDA

Features:


P
rovides
DSL
for
dealing with
time
-
based
event data:
EPL


Sliding windows: time, length, sorted, accumulating, time
-
ordering,
externally
-
timed


Continuos queries, filtering, aggregations, joins,
sub
-
queries


P
attern
detection


Events
can be plain Java objects,
XML,
java.util.Map


Input & Output
Adapters: API, JMS, Socket, HTTP, DB
-
out,
etc


Open Source EDA, Edward Yavno, Lab49

14

Esper:
API

Creating a statement and adding a listener:

EPServiceProvider

epService

=
EPServiceProviderManager.
getDefaultProvider
();

String expression =
"select
avg
(price) from
OrderEvent.win:time
(30 sec)"
;

EPStatement

statement =
epService.getEPAdministrator
().
createEPL
(expression
);

UpdateListener

myListener

=
new

UpdateListener
() {


public

void

update(
EventBean
[]
newEvents
,
EventBean
[]
oldEvents
)
{


EventBean

event =
newEvents
[0];




System.
out
.println
(
"
avg
="

+
event.get
(
"
avg
(price)"
));


}

};

statement.addListener
(
myListener
);

Sending an Event:

OrderEvent

event =
new

OrderEvent
(
"milk"
, 3.50
);

epService.getEPRuntime
().
sendEvent
(event);

Open Source EDA, Edward Yavno, Lab49

15

Esper: EPL

EPL Examples:

“select
avg
(price) from
OrderEvent.win:time
(30 sec
)”

“select
itemName
,
avg
(price)
from
OrderEvent.win:length
(100) group by
itemName


"select
itemName
, price from
OrderEvent.win:time(60
sec) where price >=
1000“

And a little more complex:

"select a.id, count(*) from pattern
[


every
a=Status
-
> (
timer:interval
(10 sec) and not Status(id=a.id
)

]
group by id"

Open Source EDA, Edward Yavno, Lab49

16

Event Flow Layers Overview

Open Source EDA, Edward Yavno, Lab49

17

Possible EDA domains


Algorithmic Trading


Real
-
time
risk analysis


Real Time web apps / social
apps


Systems Monitoring (Latency, Correct Flow,
etc
)


Fraud detection


Movements tracking (ships, trucks, couriers
, RFID chips,
etc
)


Weather patterns tracking


Medical/health monitoring


Other ideas?

Open Source EDA, Edward Yavno, Lab49

18

Thank You!

Discussion welcome!





We are giving out an
iPad
!

(email to ipad@lab49.com with the answer to "What is 49 in
binary
?“)

We are hiring!

Open Source EDA, Edward Yavno, Lab49

19

Appendix: Interesting Related Tech


Apache Camel (“Integration
Nirvana”)


ActiveMQ sub
-
project
-

Integration
framework based on
Enterprise
Integration
Patterns.
http://camel.apache.org
/



PubSubHubbub



web
-
based pub/sub protocol based
on web hooks
and Atom/RSS
http://code.google.com/p/pubsubhubbub
/



HTML5 Web Sockets


because Ajax and Comet are
just
hacks!
Standards
-
based approach to
communicating
asynchronously with the server from a
web
page
http://dev.w3.org/html5/websockets
/


Open Source EDA, Edward Yavno, Lab49

20

Appendix: Note
about
JSR107(
JCache

API)


Promising
, but stagnated (started 2001, last
known artifact in 2005 at
jsr
-
107
-
interest.dev.java.net
)


Currently not
a good fit for event listening
-

doesn't provide
element in the callback:


public
void
onPut
(Object key
); //only key, no value


Ehcache has complete implementation, also
has
good critique (at
ehcache.org/documentation/jsr107.html
)



Open Source EDA, Edward Yavno, Lab49

21