In my previous post, I introduced bowerick for easing simple Message-oriented Middleware (MoM) tasks with Clojure (and Java). In this post, I will introduce some more features of bowerick in more detail. In addition, I will discuss some of the things that, in my opinion, need further improvement.

The outline of this post is as follows:

• Motivation (Skip this if you are only interested in the technical parts.)
• Serialization Convenience
• Command Line Helpers
  • Broker
  • Client
• To-dos
  • Modularity
  • Logging
• Concluding Remarks

Motivation

In this part, I discuss my motivation for implementing the features that I will cover in this post. You can safely skip this part if you are more interested in the technical parts. However, I want to provide a bit of context, just in case it could be helpful or interesting.

As I mentioned in my previous post, I started with bowerick (or more precisely its predecessor) in scope of some research I did. Some of my needs at that time were that I wanted to ease and speed-up the prototype development and to keep my prototypes, experiments, etc. as self-contained as reasonably possible. In this scope, the intent of bowerick was to bundle the MoM related functionality and to provide easily usable abstractions on an API-level.

During my research, I experimented a bit with different ways of serializing the data that was sent via the MoM. In this context, I also considered not only the actual serialization but also compression/decompression.

Thanks to many available nice libraries, various serialization and compression methods can be easily used. However, while the implementation overhead of using a single library is quite small, the effort accumulates with the amount of libraries and serialization/compression methods that are used. Furthermore, in order to reduce the probability of errors in my experiments, I used unit tests for assessing the functionality.

Besides providing easily usable abstractions on an API-level, e.g., for serialization as mentioned above, I also considered the use of the MoM broker as another part that could be improved for easing usage and development. For this, I developed a simple command line helper runnable for easily running a simple MoM broker. After finishing my dissertation etc., I also found some time to add experimental “client” support as well.

Serialization Convenience

The default way for creating producers and consumers with bowerick is via the create-producer respectively create-consumer functions in the bowerick.jms namespace. By default the created producers/consumers use the serialization/de-serialization mechanisms of the underlying JMS transport implementation, such as ActiveMQ OpenWire.

However, depending on the used transport, the default serialization mechanisms on the producer side may not be suited for all data types. For the producer, I implemented a fallback serialization that currently defaults to serialization to UTF-8 encoded JSON via Cheshire. On the consumer side, however, it is not easily possible to identify the type of serialization that was used just based on the byte array payload (I refrained from implementing a smarter consumer as it is not possible to reliably distinguish between meta-data that identifies a serialization method and potential valid payload data that should be forwarded as is.).

In order to allow the easy use of other serialization and also compression mechanisms, I added various convenience functions for creating more specialized producers/consumers. Currently, these functions are (in the bowerick.jms namespace):

• create-nippy-producer / create-nippy-consumer
  For Nippy serialization. Below I will also show how Nippy options can be used.
• create-nippy-lzf-producer / create-nippy-lzf-consumer
  For serialization with Nippy and compression with com.ning.compress.lzf.
• create-carbonite-producer / create-carbonite-consumer
  For serialization with carbonite, which, under the hood, uses Kryo for serialization.
• create-carbonite-lzf-producer / create-carbonite-lzf-consumer
  For serialization with carbonite and compression with com.ning.compress.lzf.
• create-json-producer / create-json-consumer
  For serialization to UTF-8 encoded JSON via Cheshire.

For create-nippy-producer, below, I give some examples of possible Nippy options for compressing the data, which are set via the “normal” Nippy options map. For now, do not worry about the “1” argument. I will write about that in another post.

• (def lz4-prod (create-nippy-producer “tcp://broker:61616” “/topic/foo” 1 {:compressor taoensso.nippy/lz4-compressor}))
• (def snappy-prod (create-nippy-producer “tcp://broker:61616” “/topic/foo” 1 {:compressor taoensso.nippy/snappy-compressor}))
• (def lzma2-prod (create-nippy-producer “tcp://broker:61616” “/topic/foo” 1 {:compressor taoensso.nippy/lzma2-compressor}))

For create-nippy-consumer, no options need to be set as Nippy can identify the compression mechanisms for decompression.

From the point of interoperability and as I am primarily working with “basic” data structures, such as maps or sequences, my current favorite serialization mechanism is create-json-producer / create-json-consumer. This offers excellent interoperability across different messaging transports/protocols, such as OpenWire, STOMP, or MQTT, and also across programming languages. I plan to write about using multiple transports in a later post.

Command Line Helpers

In the following, I will describe simple broker and client related command line helper functionality provided by bowerick.

Broker

The main idea for the broker command line helper was to have an easy way for starting a simple broker instance. In the simplest form, a broker can be started with a bowerick stand-alone jar file as follows:

java -jar target/bowerick-1.99.6-standalone.jar

By default, bowerick listens sets up an OpenWire transport connector that listens on 127.0.0.1:61616. In the following listing, an example for specifying the transport/protocol is shown. Please note the additional escaped quotation marks.

java -jar target/bowerick-1.99.6-standalone.jar --url "\"stomp://127.0.0.1:61617\""

bowerick also offers convenience functionality for leveraging ActiveMQs support for multiple transports/protocols. I plan to write more about multiple transports/protocols in another blog post. For now, with respect to the broker command line helper, in the following listing, the syntax for starting a broker with multiple transports/protocols is shown.

java -jar target/bowerick-1.99.6-standalone.jar --url "[\"tcp://127.0.0.1:61616\" \"stomp://127.0.0.1:61617\"]"

Note that the list of transports is just a Clojure vector of strings in which each string defines one transport/protocol. Again, note the escaped quotation marks.

Client

While the broker command line helper essentially only provides a way for conveniently starting a MoM broker instance, the client command line helper is intended to allow more dynamic interaction. Thus, the main part of the client-side command line helper is an interactive command line interface (CLI), which uses my cli4clj library for creating the interactive CLI.

Currently, the client command line helper is in an early stage and I consider it as an experiment in progress. Use cases of the client command line helper can be, e.g.:

• simple debugging
• or serving as a playground tool for easily playing with a MoM.

In the following, I will provide a brief usage example for the client command line helper. For this example, I assume that a JMS broker is running that listens for incoming OpenWire connections on tcp://127.0.0.1:61616. Such a broker instance can be, e.g., started with the bowerick broker command line helper as follows:

java -jar target/bowerick-1.99.6-standalone.jar

The following listening shows the start of the bowerick client command line helper and the prompt of the interactive CLI:

java -jar target/bowerick-1.99.6-standalone.jar --client
...
bowerick#

In order to receive data from a destination, the “receive” command can be used as follows:

bowerick# receive tcp://localhost:61616:/topic/foo.bar
Creating JSON consumer: tcp://localhost:61616 /topic/foo.bar 1
Creating consumer: tcp://localhost:61616 /topic/foo.bar
Creating connection.
Creating destination. Type: topic Name: foo.bar
Set up consumer for: tcp://localhost:61616:/topic/foo.bar
bowerick#

The “receive” command creates a consumer that is connected to the specified broker and destination. I will discuss the “URL format” a bit more later. The consumer that is set up with the “receive” command will print the received data to stdout.

In order to send data to a destination, the “send” command can be used as follows:

bowerick# send tcp://localhost:61616:/topic/foo.bar baz
Creating JSON producer: tcp://localhost:61616 /topic/foo.bar 1
Creating producer: tcp://localhost:61616 /topic/foo.bar
Creating connection.
Creating destination. Type: topic Name: foo.bar
Sending: tcp://localhost:61616:/topic/foo.bar <- baz bowerick# Received: tcp://localhost:61616:/topic/foo.bar ->
"baz"

bowerick#

Please note that the listing also shows the reception output of the consumer that was previously set up with the “receive” command. Due to the current way of how the output is printed, the output of the consumer is simply printed on top of the prompt, which looks not so nice, in my opinion.

Also note that the producer instances created with the “send” command are cached. For subsequent invocations of “send” with an earlier used URL the producer initialization is omitted:

bowerick# send tcp://localhost:61616:/topic/foo.bar baz
Sending: tcp://localhost:61616:/topic/foo.bar <- baz bowerick# Received: tcp://localhost:61616:/topic/foo.bar ->
"baz"

bowerick#

Furthermore, the interactive CLI supports Clojure data types like maps or vectors. The default serialization mechanism uses JSON serialization via Cheshire mentioned above:

bowerick# send tcp://localhost:61616:/topic/foo.bar {"a" "string", :b [1 7 0 1]}
Sending: tcp://localhost:61616:/topic/foo.bar <- {"a" "string", :b [1 7 0 1]} bowerick# Received: tcp://localhost:61616:/topic/foo.bar ->
{"a" "string", "b" [1 7 0 1]}

bowerick#

Note that one effect of the JSON default serialization is that a keyword “:b” is changed to the string “b”.

The destination-url format is: ://

I borrowed “:/” for separating the broker part from the destination part from the way SSH SCP URLs are expressed. However, I am not yet fully convinced that this is the right way to do it. Anyway, as this is just a detail of the experimental client CLI, I do not see any major problems for changing this in the future.

To-dos

As I mentioned in this and in my previous post, bowerick and its predecessor project are a result of the needs I had during some of my experiments. Hence, I more or less “selfishly” focused primarily on the needs I had. However, with the rework I started since the fork from the original project, I also tried to get bowerick in a shape that makes it more useful for others.

Still, as this is currently a single person spare time project, there are quite a number of things that can be improved. In this section, I will briefly discuss some of these aspects.

Modularity

bowerick bundles a lot of different functionality, such as different MoM transports, serialization mechanisms, and compression methods. Thus, it has a lot of dependencies and the resulting uberjar files are pretty large.

If only a subset of the functionality offered by bowerick is used, many dependencies and space occupied in the uberjar file will be redundant. In order to counteract these issues, one potential improvement of bowerick could be to modularize bowerick into smaller better defined sub-modules.

I realize that splitting up bowerick into sub-modules is the inverse of the original intent to combine functionality and provide easily usable abstractions. However, maybe there are possibilities to keep the ease of use that is currently offered and still split bowerick into better defined sub-modules.

Logging

The logging in bowerick is currently just done via printing messages to stdout and stderr.
It is not possible to specify logging targets, such as files, or log levels to adjust the verbosity of the output. This is clearly not what one would expect in a more productive environment.

Furthermore, bowericks dependencies further complicate the situation as there may be different potentially incompatible logging mechanisms pulled in. Admittedly, I didn’t even have the time to get a complete overview of all logging mechanisms that are pulled in by the dependencies.

Consequently, providing more sophisticated logging in bowerick and to harmonizing the logging of dependencies could be another potential improvement.

Concluding Remarks

I hope that bowerick is useful for some of you. As you can see in the history, I tried to keep the dependencies as up to date as reasonably possible, up to now. Furthermore, I hope that the test-driven development approach with a high test coverage and the continuous testing and integration help to keep bowerick updated and to improve its quality. Nonetheless, there is the caveat that bowerick is a single person spare time project, for now. Constructive feedback, suggestions, etc. are always appreciated.