Java Broker Refactor

estrapadesherbetΛογισμικό & κατασκευή λογ/κού

18 Νοε 2013 (πριν από 3 χρόνια και 6 μήνες)

67 εμφανίσεις

Introduction

This document attempts to present the rationale and effects of the refactoring work applied to the Qpid Java Broker
between the Apache M2 and M3 releases.

Much of the design work for the refactoring was performed at the December ’07 Qpid Jav
a offsite held in Glasgow.
There were at least three main aims of the refactoring work. Firstly to better model the entities which exist process
messages by the broker in the end
-
to
-
end process of receiving and routing incoming messages
, enquing the mess
ages,
delivering them to consumers and then finally dequeuing. Secondly to reduce the complexity around known
programming e
rror hotspots in the existing code
-
base: in particular in the “delivery” of messages from the queue to
consumers. Thirdly to lay so
me groundwork to make the move of the broker to spe
aking new versions of the AMQP
p
ro
tocol somewhat easier by adapting the internal model in light of known developments in AMQP0
-
10.

Overall Architecture

Message Arrival:

<< Incoming Message, Exchange, Queu
eEntry, Queue diagram >>

Message Delivery:

<< Queue, QueueEntry, Subscription diagram >>


Messages

Prior to the refactoring work the notion of a “message” within the broker had been modeled by a single class
“AMQMessage”.

This object was stateful to the

extent that at various points of its lifecycle certain member
variables were expected to be “nulled” since they were no longer used


and if they were not serious memory leaks
occurred. Moreover the object was a point of lock contention as it was (potent
ially) shared by many queues which
were mutating the object from different threads. The refactoring work attempted to separate the AMQMessage into
three separate entities: the “IncomingMessage” which represents the transient state of the message as it is

arriving at
the broker and before it has been fully delivered to the destination queues; the AMQMessage which represents the
shared single instance of the message data; and the QueueEntry which represents the state of the message on an
individual queue.

I
ncomingMessage


The IncomingMessage class is used to model the message as it is being formed.
It is used to hold transient state
information which is no longer required once the message has been fully received.

In AMQP0
-
8/0
-
9 a message does not arrive at
a broker in a single atomic event. Instead the message arrives as a
sequence of events beginning with a “Basic.Publish” event which provides the information which is necessary for
routing (including the exchange to which the message is being sent, the rou
ting key to use, etc.). Following this is a
ContentHeader event. This contains header information which may be used in routing, along with properties such as
the reply
-
to address, message expiration time and the total size of the message being sent. Fol
lowing this some
number of ContentBody events are expected. The message is only complete once ContentBody data totaling the size
represented in the ContentHeader has been received.

When a Basic.Publish event is received by a Channel, an IncomingMessage ob
ject is created and attached to the
channel. If there is already an IncomingMessage attached to the channel, then there has been some error and this will
cause a failure which is relayed to the client.

The constructor of IncomingMessage simply initialize
s the object with
the data provided.

When the ContentHeader event is received this is passed to the IncomingMessage which simply stores this as a
member variable. At this point the broker has all the information which is necessary to route the message t
o the
queues to which it is to be delivered, so the route() method is called on the IncomingMessage is called. This, in turn,
invokes the route(IncomingMessage) method on the Exchange to which the message is being sent.

<< Sequence Diagram for routing >>

Future
Imp
r
o
vement:

Currently every implementation of route(IncomingMessage) in the various Exchange sub
-
classes calls
IncomingMessage.enqueue(ArrayList) as its last action. It would be better to model this by defining the
interface of Exchange as returni
ng the ArrayList and have the IncomingMessage perform the enqueue. This would
better describe the contract which IncomingMessage requires of Exchange.

It is at this point that the “mandatory” flag is tested and if there are no queues to which this message

will be delivered
then an error is created and stored as a “return message”.

Future Improvement:
AMQP0
-
10 does not use return messages to communicate errors. The broker therefore needs
to use a more protocol version independent abstraction for handling s
uch routing errors.

The “routingComplete” method is next invoked on the IncomingMessage. This method creates the “MessageHandle”
which from then on can be used to retrieve the message data (see the section on Flow
-
to
-
Disk for how this is currently
used an
d suggestions on how this should be replaced). The “routingComplete” method also takes responsibility for
ensuring that persistent messages have a record of each enqueue (to a durable queue) stored in the persistent store.

Next some number of ContentBody
events are received. As they do the “
addContentBodyFrame
” method is called.
This simply updates its record of the size of the received data, and passes the content body on to the MessageHandle.

Once all the data has been received (calculated from inspec
ting the running total of data received which is maintained
by the addContentBodyFrame method) the deliverToQueues() method on IncomingMessage is invoked. This method
creates the AMQMessage object (see below) and then uses the transactional context to del
iver the messages into the
queues which were previously calculated to be the destinations to which this message should be routed to. Note that
in order to ensure some approximation of fairness in the situation where messages are being sent to a large numb
er of
topic queues the order in which the message gets enqueued
into the list of queues is somewhat randomized.

AMQMessage


The AMQMessage class represents the single instance of the message data which is referenced by the broker once
delivery of the mes
sage to queues has completed. The main act of the refactoring has been to take functionality out
of AMQMessage and move it into either the IncomingMessage class described above, or the QueueEntry class
described below.


Ideally the AMQMessage class shoul
d provide an immutable representation of the message data (header and body)
stored within the broker. This goal has not yet been completely achieved.

Future Improvement:

remove all mutable elements from the AMQMessage


in particular move the reference
counting which is used as part of the persistent/flow
-
to
-
disk strategy out of the AMQMessage and into the store.

The AMQMessage holds references to the message handle, as well as caching the delivery flags (mandatory and
immediate), the total message size,

the expiration time for the message (when TTL is in use) and a reference to the
session in which the message was created (used when evaluating “nolocal” consumers).

Methods on the AMQMessage predominately delegate to equivalent methods on the MessageHandl
e or return the
immutable data outlined above.

Future Improvement:

the redelivered flag currently stored on the message handle should be moved to the
QueueEntry as this is a queue specific attribute and should not be shared.

QueueEntry


The QueueEntry clas
s is used to model the state of a message on a queue. Since this state is local to the queue it does
not make sense to represent it directly on the
AMQMessage. Modelling this state explicitly is one of the more
fundamental changes introduced by the refac
toring.


A QueueEntry may be either AVAILABLE, ACQUIRED, DEQUEUED or DELETED. An entry in the
AVAILABLE state is available to be sent to a consumer. In order to send to a (non
-
browsing) consumer, that
consumer needs to acquire an exclusive lock on the e
ntry. This action moves the entry to the ACQUIRED state.
Once the consumer has acquired the entry, only actions of that consumer can change the entry’s state. Either the
consumer can acknowledge the message, in which case the entry will be dequeued and
then deleted; else the
consumer may decide to release the entry (or the consumer may close before it has acknowledged) in which case the
entry is made available once more.


AVAILABLE

ACQUIRED

DEQUEUED

D
ELETED

requeue()

dequeue()

delete()

acquire()

Queue Entry State Transition Diagram


Updates to the state of a QueueEntry are always performed using a “compare
-
and
-
swa
p” operation which ensures
thread safety. For example if a consumer wishes to acquire an entry is calls acquire( ) which performs a compare
-
and
-
swap between the AVAILABLE state and the ACQUIRED state. This ensures even if two subscriptions attempt
to acq
uire the entry simultaneously, only one subscription actually succeeds in the acquisition.

Future Improvement:

Remove the unused EXPIRED state for QueueEntry.java

The QueueEntry unsurprisingly holds references to the AMQMessage and the AMQQueue on which th
e message is
enqueued. It also holds the reference to the current state of the entry (as described above) and (if there have been
any) the set of subscribers which have rejected the message (if a subscriber has previously rejected a message then
that mes
sage should not be redelivered to that subscriber).

The QueueEntry provides accessors for the queue, message and state; and provides methods to perform the desired
state transitions.

The QueueEntry also provides for a mechanism for interested observers to

be notified of state
changes for the entry. This is important where a subscription is stuck behind a large message it cannot deliver as the
message is too large for the flow control limits currently applying to the subscription.

Future Improvement:

Remov
e the unused
getDeliveredToConsumer()

method.



Queuing

The most significant change introduced by the refactoring is the alteration to the queuing model internal to the broker.
Prior to the refactoring everything possible was done to avoid messages bein
g put in an explicit queue. There were,
essentially, two distinct modes of operation “immediate” delivery and “queued” delivery


with a potentially thread
unsafe transition between them. Upon arrival at a queue the broker would look to see if any consum
er could
immediately process the message, and if so that message would be delivered to that consumer. Only if no such
consumer existed would the message be placed in an in
-
memory queue.

In order to retain some semblance of
ordering, when messages were re
turned by consumers (for instance pre
-
fetched messages returned after a consumer
has been closed) these were tacked on in a special queue at the head of any existing queue. Where selectors were in
did not meet the selection criteria. While providing exce
ptional performance this solution became difficult to reason
about. Therefore the face to face agreed that a different approach to queuing in the broker should be implemented.

Strict Ordering


The fundamental principal of the new Queuing model is that the

queue provides a strict order on the messages being
enqueued. Furthermore that order is maintained through the lifetime of the entries on the queue: thus if a
message
is
returned (e.g. the prefetched messages being released upon the consumer closing) the

order of that message with
respect to other messages on the queue is maintained.


The strict ordering is enforced by the use of a queue data
-
structure. In order for this to be performant, the data
structure uses a lockless thread
-
safe designed based aro
und the same algorithm used in the
java.util.concurrent.ConcurrentLinkedList (more precisely it is based on the public domain implementation in the
backport util concurrent project). See the section on Concurrent List implementations for more details.

Ea
ch subscription keeps a “pointer” into the list denoting the point at which that particular subscription has reached.
A particular subscription will only deliver a message if it is the next AVAILABLE entry on the queue after the
pointer which it maintains

which matches any selection criteria the subscription may have.

Thread safety is maintained by using the thread
-
safe atomic compare
-
and
-
swap operations for maintaining queue
entry state (as described above) and also for updating the pointer on the subscri
ption. The queue is written so that
many threads may be simultaneously attempting to perform deliveries simultaneously on the same messages and/or
subscriptions.

Queue

HEAD

TAIL

Subscription 1


Subscription 2


Subscription 3



Enqueing


When a message is enqueued (using the enqueue() method on the AMQQueue implementati
on) it is first added to the
tail of the list. Then the code iterates over the subscriptions (starting at the last subscription the queue was known to
have delivered for reasons of fairness). For each subscription found it attempts delivery (details desc
ribe below). If
the message cannot be delivered to any subscription then
the “immediate” flag on the message is inspected. If the
message required immediate delivery then
the message is immediately dequeued, otherwise
an asynchronous job is
created to at
tempt delivery at a later point.

(Note there is a “shortcut” path for queues which have an exclusive subscriber. In this case we know there is one and
only one subscriber and so we can go directly to trying to deliver to it without worrying about iterator
s, etc.)


Potential Issue:
Looking at the code which performs the check of the immediate flag I believe there is a race
condition:



if (entry.immediateAndNotDelivered())


{


dequeue(storeContext, entry);


entry.dispose
(storeContext);


}

This does not look to be safe in the case where there is a simultaneous execution of an asynchronous deliver which
may acquire the message between the check of immediateAndDelivered and dequeue. Instead of calling dequeue
directl
y we should instead
do a safe compare
-
and
-
swap test to make sure the entry state is “AVAILABLE” before
setting it to DEQUEUED. The implementation of this should probably look much like the implementation of
entry.dequeue except for the different expected
starting state.

Immediate Delivery


For each subscription to the queue, we call the following code:


private void deliverToSubscription(final Subscription sub, final QueueEntry entry)


throws AMQException


{



sub.getSendLock();



try


{


if (subscriptionReadyAndHasInterest(sub, entry)


&& !sub.isSuspended())


{


if (!sub.wouldSuspend(entry))


{


if (!sub.isBrowser() && !entry.acquire(s
ub))


{


// restore credit here that would have been taken away by wouldSuspend
since we didn't manage


// to acquire the entry for this subscription


sub.restoreCredit
(entry);


}


else


{



deliverMessage(sub, entry);



}


}


}


}


finally


{


sub.releaseSendLock();


}


}


This code first takes a lock on the subscriber (this prevents it being removed while we are carrying out this operation).
It then tests if the given subscription
can take this message at the moment (see below for more details). It then
further
tests if there is enough flow control credit to send this message to this subscription. If there is credit (and the
subscription is not a “browser” then is attempts to acquire the entry ( entry.acquire(sub) ). If the acquisition is
successful (or

if the subscription is a browser and thus does not need to acquire the entry) then the entry is delivered
to the subscription, else the credit that would have been used by the message if sent is restored.

The most interesting method called in the above is

subscriptionReadyAndHasInterest(sub, entry)
:


private boolean subscriptionReadyAndHasInterest(final Subscription sub, final QueueEntry
entry)


{


// We need to move this subscription on, past entries which are already acquired, or
deleted or
ones it has no


// interest in.


QueueEntry node = sub.getLastSeenEntry();


while (node != null && (node.isAcquired() || node.isDeleted() || !sub.hasInterest(node)))


{



QueueEntry newNode = _entries.next(node);



if (newNode != null)


{


sub.setLastSeenEntry(node, newNode);


node = sub.getLastSeenEntry();


}


else


{


node = null;


break;


}




}



if (node == entry)


{


// If the first entry that subscription can process is the one we are trying to
deliver to it, then we are


// good


return true;


}


else


{


return false;


}



}


Here we see how the subscription is inspected to see where its pointer into the queue (the last seen entry) is in respect
to the entry we are tr
yi
ng to deliver. We start from the subscription’s current lastSeenEntry and wor
k our way down
the list passing over entries which are already acquired by other subscriptions, deleted, or which this subscription has
no interest in (e.g. because the node does not meet the subscription’s selection criteria)
; all the while we can update
the lastSeenEntry to take it past the entries this subscription has now inspected
. Performing this iteration we will
eventually arrive at the next entry the subscription is interested in (or just fall off the end of the list).

At this point
either the ne
xt entry that the subscription is interested in is the entry we wish to deliver (success!) or not.



Asynchronous Delivery


If there are no subscriptions that can currently take delivery of the message then we need to schedule an asynchronous
delivery.

Wh
ile the code is thread safe and could cope with multiple threads performing asynchronous delivery
simultaneously, we limit ourselves to only having one asynchronous delivery job scheduled at any one time, so as not
to overwhelm the broker:


public void
deliverAsync()


{


_stateChangeCount.incrementAndGet();



Runner runner = new Runner();



if (_asynchronousRunner.compareAndSet(null, runner))


{


_asyncDelivery.execute(runner);


}


}


Here we first incr
ement our count of “stateChanges”. This provides us with a way of knowing between loops of the
asynchronous delivery thread whether anything else has happened that makes it worth our while running the
asynchronous delivery loop again (in effect it prevent
s us having to always add another thread to cope with race
conditions where we want to start the async delivery just as it is ending). We then create a new instance of the
asynchronous delivery “Runner”, and attempt to make this instance the current one b
y means of the ubiquitous
compare
-
and
-
swap operation. Here we test if we are the thread that moved the queue from having no asynchronous
runner to having one; and if so we need to schedule the runner to execute by way of calling
_asyncDelivery.execute(run
ner).


The actual work of the asynchronous delivery is done in the

processQueue(Runnable runner)
method.




private void processQueue(Runnable runner) throws AMQException


{


long stateChangeCount;


long previousStateChangeCount = Lon
g.MIN_VALUE;


boolean deliveryIncomplete = true;



int extraLoops = 1;


int deliveries = MAX_ASYNC_DELIVERIES;



_asynchronousRunner.compareAndSet(runner, null);



while (deliveries != 0 &&


((previousState
ChangeCount != (stateChangeCount = _stateChangeCount.get())) ||
deliveryIncomplete)


&& _asynchronousRunner.compareAndSet(null, runner))


{


// we want to have one extra loop after every subscription has reached the point w
here
it cannot move


// further, just in case the advance of one subscription in the last loop allows a
different subscription to


// move forward in the next iteration



if (previousStateChangeCount != stateChangeCount)



{


extraLoops = 1;


}



previousStateChangeCount = stateChangeCount;


deliveryIncomplete = _subscriptionList.size() != 0;


boolean done = true;


In this first fragment of the method we se
e the constraint on how long the asynchronous delivery will keep attempting
to deliver more messages.


The first constraint “deliveries != 0” is testing a countdown value “deliveries” which is intialised with an initial
maximum (currently set to 10): eve
ry successful delivery the thread makes decrements this counter. This implements
a limit on how long the processQueue method will be allowed to run for, stopping this queue from starving other
queues of processor time. At the end, if this countdown was t
he factor to cause the loop to terminate, the
asynchronous delivery is scheduled to run again.


The second constraint “
((previousStateChangeCount != (stateChangeCount = _stateChangeCount.get())) ||
deliveryIncomplete)

” is testing whether there is provably

nothing left to do on this queue. The first half tests if
there have been any changes since the last iteration that have incremented that state change count (and thus require
another loop), the second half says, “even if there haven’t been any changes ke
ep looping if last time round we
thought there was still more to do”.


The final constraint “
_asynchronousRunner.compareAndSet(null, runner))
” is our familiar compare
-
and
-
swap
operation ensuring that this is the designated instance of the asynchronous proc
esser running.



This loop runs, attempting to deliver one message in each iteration:




SubscriptionList.SubscriptionNodeIterator subscriptionIter =
_subscriptionList.iterator();


//iterate over the subscribers and try to advance th
eir pointer


while (subscriptionIter.advance())


{


boolean closeConsumer = false;


Subscription sub = subscriptionIter.getNode().getSubscription();


if (sub != null)


{


Ite
rate over the subscriptions on the queue…



sub.getSendLock();


Lock the subscription so it does not get deleted while attempting to delvier to it.



try


{


QueueEntry node =
moveSubscriptionToNextNode(sub);


Find the next node which the subscription should try to deliver by skipping over already acquired entries, if it is null
then this subscription is at the tail of the queue.



if (node != null && sub.
isActive())


{

Keep a track of whether this subscription is really active and whether we managed to advance the pointer on this
subscription in this loop (these values go into determining if there is anything left to do in a new loop
).



boolean advanced = false;


boolean subActive = false;



if (!(node.isAcquired() || node.isDeleted()))


{


if (!s
ub.isSuspended())


{

The node is not yet acquired or deleted, and we can now be sure the subscription is active.


subActive = true;


if (sub.hasInterest(no
de))


{

The following code is similar to that in the deliverToSubscription method described previously. It should be possible
to factor this out.

The primary difference is the behaviour with a browser where need to expl
icitly note that we have
advanced.



if (!sub.wouldSuspend(node))


{


if (!sub.isBrowser() && !node.acquire(sub))



{


sub.restoreCredit(node);



}


else


{



deliverMessage(sub, node);


deliveries
--
;



if (sub.isBrowser())


{



QueueEntry newNode = _entries.next(node);



if (newNode != null)


{



sub.setLastSeenEntry(node, newNode);


node = sub.getLastSeenEntry();


advanced = true;


}



}


}


done = false;


}


else // Not enoug
h Credit for message and wouldSuspend


{


This case covers the scenario where we are using bytes based flow control, and the currently available credit is less
than the size of the next message. We need to wait eithe
r for the credit to be increased (which will cause a state
change event) or the entry to be picked off by another subscription (which we capture with the state change listener)



//QPID
-
1187
-

Treat the subscripti
on as suspended for
this message


// and wait for the message to be removed to continue
delivery.


subActive = false;



node.ad
dStateChangeListener(new
QueueEntryListener(sub, node));


}


}


else


{


// this subscription is not interested in this node so we
can skip over it


QueueEntry newNode = _entries.next(node);


if (newNode != null)



{


sub.setLastSeenEntry(node, newNode);


}


}


}


}

Here we calculate if there
is anything left to do on this particular subscription. If we are at the tail of the subscription,
or the subscription is no longer active, then this subscription can be considered done. If all subscriptions are done
then we are truly finished.




final boolean atTail = (_entries.next(node) == null);


done = done && (!subActive || atTail);


Here calculate if we need to auto
-
close the subscription


we do this if we are at the tail of the queue, and we d
idn’t
advance in this iteration and this is an auto
-
close subscription.



closeConsumer = (atTail && !advanced && sub.isAutoClose());


}


}


finally



{


sub.releaseSendLock();


}



if (closeConsumer)


{


unregisterSubscription(sub);



ProtocolOutputConverter converter =
sub.getC
hannel().getProtocolSession().getProtocolOutputConverter();


converter.confirmConsumerAutoClose(sub.getChannel().getChannelId(),
sub.getConsumerTag());


}



}



This ends the iteration over subscript
ions, now we calculate if we believe there we should try iterating over the
subscriptions again. We use the value of “done” we calculated while iterating over the subscriptions to determine if
we need to loop again. If we believe we are done and we have
already used our “extra” loop then we can stop. If we
are “done” but we have not yet used the extra loop, then we decrement the extra loop counter (setting it to 0 might be
clearer since 0 and 1 are the only valid values) and go round one more time. If w
e are not done then we restore
extraLoops to 1.



if (done)


{


if (extraLoops == 0)


{


deliveryIncomplete = false;


}


else



{


extraLoops
--
;


}


}


else


{


extraLoops = 1;


}


}


_asynchronousRunner.set(null);



}



This ends the the “while(…” loop. The final action in the asynchronous process is to determine if we need to
schedule ourselves for another execution:




// If deliveries == 0 then the limitting factor was the time
-
slicing rather than availabl
e
messages or credit


// therefore we should schedule this runner again (unless someone beats us to it :
-
) ).


if (deliveries == 0 && _asynchronousRunner.compareAndSet(null, runner))


{


_asyncDelivery.execute(runner);



}


}




Subscriptions


Subscription model the entities created by the receiving of a “Basic.Consume” event in AMQP0
-
8/0
-
9. That is they
represent a relationship between an AMQP Channel (equivalent to a Java JMS Session) and a queue. As messages
ar
e placed on the queue, the queue takes responsibility for as quickly as possible finding a subscriber which is willing
to take the message. The subscriber is responsible for delivering the message to the receiving client. As outlined
above, a significant

change introduced by the refactoring is that the Subscriptions now maintain state representing a
pointer into the queue. This pointer represents the current position where the subscription can guarantee that no
message prior to that is of interest to it.

Generally this pointer only ever moves forward through the queue (see the
section on reject and release for the exception to this rule). This is the only dynamic state maintained directly by the
subscription.

Different subclasses of SubscriptionImpl are

used to model the different behviour associated with different
acknowledgement modes. The subclasses used are AckSubscription, NoAckSubscription, BrowserSubscription and
GetNoAckSubscription.

The last of these is a special implementation which is used t
o model a Basic.Get command
as a temporary subscription that can only ever receive one message. Modelling Get in this way mirrors how the same
semantics are implemented in 0
-
10 and removes having two separate ways to dequeue messages from the queue.

Addin
g


When a “Basic.Consume” event is processed the subscription is added to the list of subscriptions on the queue, the
“pointer” in the subscription is set to point at the head of the list of queue entries, and then an asynchronous job is
kicked off to deli
ver to that subscription as many messages as can be delivered starting at the head. This uses an
algorithm almost identical to that described above to asynchronous message delivery, except it only considers the one
subscription. This is found within the
“flushSubscription” method of the queue (flushing a subscription is a 0
-
10
concept where you attempt to send as much as possible to a given subscription and then signal completion when
either the subscription’s credit runs out or there are no more messages

on the queue).

Future Improvement:

Factor out the common code between
flushSubscription and processQueue.

Removal


A consumer is removed either through the reception of a “Basic.Cancel” event or through the closure of the
encapsulating channel.

For threa
d safety, the first action is to remove the subscription from the list of subscriptions
that the asynchronous delivery task iterates over. Next the subscription’s close() method is called. This takes out a
lock on the subscription (to avoid conflicting w
ith any attempt to concurrently send to the subscription) and changes
the subscription’s state to “closed”. The combination of these steps allow us to assert that after that point in time the
subscription will not be used by any other threads to attempt t
o deliver messages. Next the subscription’s pointer into
the queue is null
-
ed out in a thread
-
safe way


this is done to prevent memory leaks due to references being held to
points in the queue (due to the way that the concurrent
-
safe queues work “deleted
” elements may not be eligible for
garabage collection for some time).

Finally, if the queue is of “auto
-
delete” type and the subscription being removed is the last subscription attached to
the queue, then the queue needs to be deleted.

Flow Control


In co
ntrast to the handling of flow control / QoS in the prior versions of the broker, after the refactoring there are now
concrete classes modeling the behaviour of the flow control algorithm. Moreover these flow control managers are set
at the subscription l
evel rather than only having one instance for the entire channel. For AMQP 0
-
8 and 0
-
9 flow
control still happens at a per
-
channel level, so the same instance of the flow control manager is shared between all
subscriptions on a channel. For 0
-
10 implemen
tations we will be able to use the same code to implement the per
-
subscription flow
-
control model that it utilizes.

Acknowledgement

Reject and Release


Messages delivered to a subscription may subsequently be returned to the queue either explicitly (by use

of a reject
command) or implicitly (by the closure of the channel). In this case the message must be made available again to
subscribers to the queue. The issue here is that the pointers held by the subscriptions are likely to be in advance of
the point

to which the message is being returned. Thus for each message that is returned we must iterate over all
subscribers to the queue, and if their current pointer is in advance of the returned message it must be moved back
such that the next entry that that
subscriber sees is the returned message. We do not reset the pointer for browsing
consumers however as doing so would lead to all the browsed messages that are after the returned message in the
queue being redelivered to the browsing subscription.


Prior
ity Queues

The fundamental difference between Priority Queues and other Queues is that the strict ordering on the queue is not
purely FIFO. Instead the ordering is a combination of FIFO and the priority assigned to the message. To provide
strict priority

ordering (where a message of higher priority will
always

be delivered in preference to a message of
lower priority) we can implement

a priority queue as an ordered list of standard sub
-
queues with the ordering between
them defined such the tail of the hig
hest priority sub
-
queue is followed by the head of the sub
-
queue of the next
highest priority.

By defining the standard queue implementation such that the methods which determine the ordering between the
nodes can be overridden, the implementation of such
a strict priority queue is almost trivial.



The interface QueueEntryList provides an extension point for adding a new queue implementation type:

public interface QueueEntryList

{


AMQQueue getQueu
e();



QueueEntry add(AMQMessage message);



QueueEntry next(QueueEntry node);



QueueEntryIterator iterator();



QueueEntry getHead();

}


The class PriorityQueueList provides the concrete implementation of a strict priority queue as defined ab
ove. The
constructor takes an argument defining how many priority levels are to be provided.

Priority
Queue


TAIL

Subscription 2




HEAD


Subscription 1


Subscription 3



Subqueue
Priority


When a message is added to the list by calling the add() method, the class first works out which sub
-
queue to add the
message to. This is determined by an algo
rithm identical to that defined in the AMQP0
-
10 specification and
comp
liant with the JMS requirements. The message is added to the tail of the appropriate sub
-
queue.

The next() method returns the QueueEntry which logically follows the QueueEntry provided
by the caller. First we
can simply look at the sub
-
queue in which the passed QueueEntry is actually in. If there is a subsequent entry in that
sub
-
queue then we use that. If there is no subsequent entry in the sub
-
queue then we must find the next highes
t
priorty subqueue and take the head of that (repeating until we find a subqueue which is non
-
empty).

The getHead() method iterates over the subqueues to find the highest priority sibqueue which is non
-
empty and then
returns the head of that subqueue.

The
iterator() method returns an iterator that respects the ordering defined above.

The only other difference between a PriortyQueue and the standard queue is that new messages arriving may be
logically “before” messages that have arrived previously (i.e. a hi
gh priority message is always logically prior to a
low priority message in the queue). This means that on arrival of a message into the queue all subscriptions need to
be inspected to make sure their pointer is not “ahead” of the new arrival.

Thus the ent
ire implementation of AMQPriorityQueue is as follows:

public class AMQPriorityQueue extends SimpleAMQQueue

{


protected AMQPriorityQueue(final AMQShortString name,


final boolean durable,


fina
l AMQShortString owner,


final boolean autoDelete,


final VirtualHost virtualHost,


int priorities)


throws AMQException


{


super(name, durable,

owner, autoDelete, virtualHost, new
PriorityQueueList.Factory(priorities));


}



public int getPriorities()


{


return ((PriorityQueueList) _entries).getPriorities();


}



@Override


protected void checkSubscriptionsNotAheadOfDeli
very(final QueueEntry entry)


{


// check that all subscriptions are not in advance of the entry


SubscriptionList.SubscriptionNodeIterator subIter = _subscriptionList.iterator();


while(subIter.advance() && !entry.isAcquired())



{


final Subscription subscription = subIter.getNode().getSubscription();


QueueEntry subnode = subscription.getLastSeenEntry();


while(subnode != null && entry.compareTo(subnode) < 0 && !entry.isAcquired())



{


if(subscription.setLastSeenEntry(subnode,entry))


{


break;


}


else


{


subnode = subscription.getLastSeenEntry();


}



}



}


}


}


The constructor merely ensures passes up the machinery to ensure a PriorityQueueList (as described above) is used
for the underlying queueing model. The getPriorities() method is overridden by delegating to the PriorityQueue
List
and then the algorithm for updating the subscriptions’ pointers into the queue is implemented in
checkSubscriptionsNotAheadOfDelivery
. Thread
-
safe compare
-
and
-
swap operations are used to update the pointer
in
-
case other threads are also trying to mov
e it; and the loop terminates early if the new QueueEntry has already been
acquired.