Apache Tribes Development
|What is Tribes|
Tribes is a messaging framework with group communication abilities. Tribes allows you to send and receive
messages over a network, it also allows for dynamic discovery of other nodes in the network.
And that is the short story, it really is as simple as that. What makes Tribes useful and unique will be
described in the section below.
The Tribes module was started early 2006 and a small part of the code base comes from the clustering module
that has been existing since 2003 or 2004.
The current cluster implementation has several short comings and many work arounds were created due
to the complexity in group communication. Long story short, what should have been two modules a long time
ago, will be now. Tribes takes out the complexity of messaging from the replication module and becomes
a fully independent and highly flexible group communication module.
In Tomcat the old
modules/cluster has now become
modules/ha (replication). This will allow development to proceed and let the developers
focus on the issues they are actually working on rather than getting boggled down in details of a module
they are not interested in. The understanding is that both communication and replication are complex enough,
and when trying to develop them in the same module, well you know, it becomes a cluster :)
Tribes allows for guaranteed messaging, and can be customized in many ways. Why is this important?
Well, you as a developer want to know that the messages you are sending are reaching their destination.
More than that, if a message doesn't reach its destination, the application on top of Tribes will be notified
that the message was never sent, and what node it failed.
|Why another messaging framework|
I am a big fan of reusing code and would never dream of developing something if someone else has already
done it and it was available to me and the community I try to serve.
When I did my research to improve the clustering module I was constantly faced with a few obstacles:
1. The framework wasn't flexible enough
2. The framework was licensed in a way that neither I nor the community could use it
3. Several features that I needed were missing
4. Messaging was guaranteed, but no feedback was reported to me
5. The semantics of my message delivery had to be configured before runtime
And the list continues...
So I came up with Tribes, to address these issues and other issues that came along.
When designing Tribes I wanted to make sure I didn't lose any of the flexibility and
delivery semantics that the existing frameworks already delivered. The goal was to create a framework
that could do everything that the others already did, but to provide more flexibility for the application
developer. In the next section will give you the high level overview of what features tribes offers or will offer.
To give you an idea of the feature set I will list it out here.
Some of the features are not yet completed, if that is the case they are marked accordingly.
Tribes is built using interfaces. Any of the modules or components that are part of Tribes can be swapped out
to customize your own Tribes implementation.
In the default implementation of Tribes uses TCP for messaging. TCP already has guaranteed message delivery
and flow control built in. I believe that the performance of Java TCP, will outperform an implementation of
Java/UDP/flow-control/message guarantee since the logic happens further down the stack.
Tribes supports both non-blocking and blocking IO operations. The recommended setting is to use non blocking
as it promotes better parallelism when sending and receiving messages. The blocking implementation is available
for those platforms where NIO is still a trouble child.
Different Guarantee Levels
There are three different levels of delivery guarantee when a message is sent.
You can of course write even more sophisticated guarantee levels, and some of them will be mentioned later on
in the documentation. One mentionable level would be a 2-Phase-Commit, where the remote applications don't receive
the message until all nodes have received the message. Sort of like a all-or-nothing protocol.
- IO Based send guarantee. - fastest, least reliable
This means that Tribes considers the message transfer to be successful
if the message was sent to the socket send buffer and accepted.
On blocking IO, this would be
On non blocking IO, this would be
socketChannel.write(), and the buffer byte buffer gets emptied
followed by a
socketChannel.read() to ensure the channel still open.
read() has been added since
write() will succeed if the connection has been "closed"
when using NIO.
- ACK based. - recommended, guaranteed delivery
When the message has been received on a remote node, an ACK is sent back to the sender,
indicating that the message was received successfully.
- SYNC_ACK based. - guaranteed delivery, guaranteed processed, slowest
When the message has been received on a remote node, the node will process
the message and if the message was processed successfully, an ACK is sent back to the sender
indicating that the message was received and processed successfully.
If the message was received, but processing it failed, an ACK_FAIL will be sent back
to the sender. This is a unique feature that adds an incredible amount value to the application
developer. Most frameworks here will tell you that the message was delivered, and the application
developer has to build in logic on whether the message was actually processed properly by the application
on the remote node. If configured, Tribes will throw an exception when it receives an ACK_FAIL
and associate that exception with the member that didn't process the message.
Per Message Delivery Attributes
Perhaps the feature that makes Tribes stand out from the crowd of group communication frameworks.
Tribes enables you to send to decide what delivery semantics a message transfer should have on a per
message basis. Meaning, that your messages are not delivered based on some static configuration
that remains fixed after the message framework has been started.
To give you an example of how powerful this feature is, I'll try to illustrate it with a simple example.
Imagine you need to send 10 different messsages, you could send the the following way:
As you can imagine by now, these are just examples. The number of different semantics you can apply on a
per-message-basis is almost limitless. Tribes allows you to set up to 28 different on a message
and then configure Tribes to what flag results in what action on the message.
Message_1 - asynchronous and fast, no guarantee required, fire and forget
Message_2 - all-or-nothing, either all receivers get it, or none.
Message_3 - encrypted and SYNC_ACK based
Message_4 - asynchronous, SYNC_ACK and call back when the message is processed on the remote nodes
Message_5 - totally ordered, this message should be received in the same order on all nodes that have been
send totally ordered
Message_6 - asynchronous and totally ordered
Message_7 - RPC message, send a message, wait for all remote nodes to reply before returning
Message_8 - RPC message, wait for the first reply
Message_9 - RPC message, asynchronous, don't wait for a reply, collect them via a callback
Message_10- sent to a member that is not part of this group
Imagine a shared transactional cache, probably >90% are reads, and the dirty reads should be completely
unordered and delivered as fast as possible. But transactional writes on the other hand, have to
be ordered so that no cache gets corrupted. With tribes you would send the write messages totally ordered,
while the read messages you simple fire to achieve highest throughput.
There are probably better examples on how this powerful feature can be used, so use your imagination and
your experience to think of how this could benefit you in your application.
Interceptor based message processing
Tribes uses a customizable interceptor stack to process messages that are sent and received.
So what, all frameworks have this!
Yes, but in Tribes interceptors can react to a message based on the per-message-attributes
that are sent runtime. Meaning, that if you add a encryption interceptor that encrypts message
you can decide if this interceptor will encrypt all messages, or only certain messages that are decided
by the applications running on top of Tribes.
This is how Tribes is able to send some messages totally ordered and others fire and forget style
like the example above.
The number of interceptors that are available will keep growing, and we would appreciate any contributions
that you might have.
Threadless Interceptor stack
The interceptor don't require any separate threads to perform their message manipulation.
Messages that are sent will piggy back on the thread that is sending them all the way through transmission.
The exception is the
MessageDispatchInterceptor that will queue up the message
and send it on a separate thread for asynchronous message delivery.
Messages received are controlled by a thread pool in the
The channel object can send a
heartbeat() through the interceptor stack to allow
for timeouts, cleanup and other events.
MessageDispatchInterceptor is the only interceptor that is configured by default.
Tribes support parallel delivery of messages. Meaning that node_A could send three messages to node_B in
parallel. This feature becomes useful when sending messages with different delivery semantics.
Otherwise if Message_1 was sent totally ordered, Message_2 would have to wait for that message to complete.
Through NIO, Tribes is also able to send a message to several receivers at the same time on the same thread.
Silent Member Messaging
With Tribes you are able to send messages to members that are not in your group.
So by default, you can already send messages over a wide area network, even though the dynamic discover
module today is limited to local area networks by using multicast for dynamic node discovery.
Of course, the membership component will be expanded to support WAN memberships in the future.
But this is very useful, when you want to hide members from the rest of the group and only communicate with them