Home | Ciber
Knowledge page of Ciber Netherlands

Archive for December, 2011

Enterprise Integration: ActiveMQ network of brokers


ActiveMQ is a powerful and widely used open source messaging server, more commonly known as "message broker". It’s being used often in Java based enterprise integration solutions, and provides "Enterprise features" like clustering, multiple message stores, and ability to use any database as a JMS persistence provider besides VM, cache, and journal persistency. For more information on ActiveMQ, visit the website [1] and check the documentation. In general, JMS message brokers often are the heart of (enterprise) integration solutions.


There is a commercially supported version available at FuseSource[2], as part of the FuseSource stack consisting of an Enterprise Service Bus (based on ServiceMix), Message Broker (based on ActiveMQ), Mediation Router (based on Apache Camel) and web services stack (based on CXF).

Besides these products, FuseSource offers other products such as Fuse HQ, an infrastructure monitoring tool and FuseIDE, an eclipse base visual development tool for designing Camel routes.


Concept: Network of Brokers

One of the more interesting features of ActiveMQ is the ability to create a network of brokers, thus connecting and clustering multiple instances of the message broker on different servers in a network (or network of networks, on the internet, or somewhere in the cloud)

Networks of message brokers [3] in ActiveMQ work quite differently than more familiar models such as that of physical networks. In this article I will try to explain each component piece progressively moving up in complexity from a single broker through to a full blown network. At the end you should have basic understanding of how these networks behave and be able to reason about the interactions across different topologies. Please note that this is not intended as an ActiveMQ tutorial. Thus, no code examples or configuration snippets this time…


Concept: Producer-Consumer

One of the key things in understanding broker-based messaging is that the production, or sending of a message, is disconnected from the consumption of that message. The broker acts as an intermediary, serving to make the method by which a method is consumed as well as the route that the message has travelled orthogonal to its production. You don’t really need to understand the entire postal system to know that when you post a letter in the big orange PostNL (or whatever it is called today) box, it eventually will arrive in the little box at the front of the recipient’s house. Same idea applies here.


Producer and consumer are unaware of each other; only the broker they are connected to.

Connections are shown in the direction of where they were established from (i.e. Consumer connects to Broker).

Out of the box when a standard message is sent to a queue from a producer, it is sent to the broker, which persists it in its message store. By default this is in KahaDB, but it can configured to be stored in memory, which buys performance at the cost of reliability. Once the broker has confirmation that the message has been persisted in the journal (the terms journal and message store are often used interchangeably), it responds with an acknowledgement back to the producer. The thread sending the message from the producer is blocked at this time.


On the consumption side, when a message listener is registered or a call to receive() is made, the broker creates a subscription to that queue. Messages are fetched from the message store and passed to the consumer; it’s usually done in batches, and the fetching is a lot more complex than simply read from disk, but that’s the general idea.

The consumer will usually at this stage process the message and subsequently acknowledge that the message has been consumed. The broker then updates the message store marking that message as consumed, or just deleting it (this depends on the persistence mechanism).


So what happens when there are more than one consumer on a queue? All things being equal, and ignoring consumer priorities, the broker will in this case hand out incoming messages in a round-robin manner to each subscriber.



Concept: Store-and-forward

Now let’s scale up to two brokers, Broker1 and Broker2. In ActiveMQ a network of brokers is set up by connecting a networkConnector to a transportConnector (think of it as a socket listening on a port). A networkConnector is an outbound connection from one broker to another.


When a subscription is made to a queue on Broker2, that broker tells the other brokers that it knows about (in our case, just Broker1) that it is interested in that queue; another subscription is now made on Broker1 with Broker2 as the consumer. As far as an ActiveMQ broker is concerned there is no difference between a standard client that consumes messages and another broker acting on behalf of a client. They are treated in exactly the same manner.

So now that Broker1 sees a subscription from Broker2, what happens? The result is a hybrid of the two producer and consumer behaviors. Broker1 is the producer, and Broker2 the consumer. Messages are fetched from Broker1?s message store, passed to Broker2. Broker2 processes the message by “store”-ing it in its journal, and acknowledges consumption of that message. Broker1 then marks the message as consumed.


The simple consume case applies as Broker2 “forwards” the message to its consumers, as if the message was produced directly into it. Neither the producer nor consumer are aware that a network of brokers exists, it is orthogonal to their functionality – a key driver of this style of messaging.

Concept: Local and remote consumers

It has already been noted that as far as a broker is concerned, all subscriptions are equal. To it there is no difference between a local “real” consumer, and another broker that is going to forward those messages on. Hence incoming messages will be handed out round-robin as usual. In case of 2 consumers (Consumer1 on Broker1, and Consumer2 on Broker2) if messages are produced to Broker1, both consumers will each receive the same number of messages.


A networkConnector is unidirectional by default, which means that the broker initiating the connector acts as a client, forwarding its subscriptions. Broker2 in this case subscribes on behalf of its consumers to Broker1. Broker2 however will not be made aware of subscriptions on Broker1. networkConnectors can however be made duplex, such that subscriptions are passed in both directions.

So let’s take it one step further with a network that demonstrates why it is a bad idea to connect brokers to each other in an ad-hoc manner. Let’s add Broker3 into the mix such that it connects into Broker1, and Broker2 sets up a second networkConnector into Broker3. All networkConnectors are set up as duplex.


This is a common approach people take when they first encounter broker networks and want to connect a number of brokers to each other, as they are naturally used to the internet model of network behaviour where traffic is routed down the shortest path. If we think about it from first principles, it quickly becomes apparent that is not the best approach.

Let’s examine what happens when a consumer connects to Broker2.

  • Broker2 echoes the subscription to the brokers it knows about - Broker1 and Broker3.
  • Broker3 echoes the subscription down all networkConnectors other than the one from which the request came; it subscribes to Broker1.
  • A producer sends messages into Broker1.
  • Broker1 stores and forwards messages to the active subscriptions on its transportConnector; half to Broker2, and half to Broker3.
  • Broker2 stores and forwards to its consumer.
  • Broker3 stores and forwards to Broker2.

Eventually everything ends up at the consumer, but some messages ended up needlessly travelling Broker1->Broker3->Broker2, while the others went by the more direct route Broker1->Broker2. Add more brokers into the mix, and the store-and-forward traffic increases exponentially as messages flow through any number of weird and wonderful routes.


The message flow overview above shows lots of unnecessary store-and-forward.

Fortunately, it is possible to avoid this by employing other topologies, such as hub and spoke.


Better, isn’t it? A message can flow between any of the numbered brokers via the hub and a maximum of 3 hops.

You can also use a more nuanced approach that includes considerations such as unidirectional networkConnectors that pass only a certain subscriptions, or reducing consumer priority such that further consumers have a lower priority than closer ones.

Each network design needs to be considered separately and trades off considerations such message load, amount of hardware at your disposal, latency (number of hops) and reliability. When you understand how all the parts fit and think about the overall topology from first principles, it’s much easier to work through.


[1] Apache ActiveMQ : http://activemq.apache.org

[2] FuseSource : http://www.fusesource.com

[3] Network of brokers : http://activemq.apache.org/networks-of-brokers.html

No comments