## Slides of my :clojureD Talk #clojuredconf

I uploaded the slides for my :clojureD talk:

Packet Capturing with the JVM and Clojure – Yes, we can!

Huge thanks to the organizers, speakers, attendees, and everyone else involved in :clojureD! 🙂

## Note to Self: Clojure Code in LaTeX vs. Copy&Paste

This post is a somewhat extended “Note to Self” in the sense that some of this may also hopefully be useful for others.

I like LaTeX a lot and its “listings” package is great for including source code in documents. However, while the source code typically looks good in the generated documents, there are problems when copying and pasting the code from the listings.

Typically, one of the reasons of providing a code example is that someone can reproduce certain things. Hence, it would be great if it was possible to directly copy and paste code from listings generated with LaTeX to a source code file or REPL. Unfortunately, there are some issues related to this.

At first, I will show a minimum working example for illustrating the problem. Afterwards, I show a minimum working example with the solution I use so far. The current “solution” is based on various things I could find on the Internet and merging all the bits and pieces. Actually, when searching for “LaTeX” and “lstlisting” and “copy paste” or “source code” one can find many posts, questions, etc. with all sorts of answers.

The Problem

On my machine, the following “naive” LaTeX document

\documentclass{scrartcl}

\usepackage{listings}

\begin{document}

\begin{lstlisting}
(let [my-list '(1 2 3)
my-product (reduce * my-list)]
(println "My-Product (1*2*3): " my-product))
\end{lstlisting}

\end{document}


produces the output shown in the following screenshot:

The problem is that copying and pasting this from a PDF viewer, such as Evince, results in the following when pasted:

( l e t [ my− l i s t ’ ( 1 2 3 )
my−p r o d u c t ( r e d u c e ∗ my− l i s t ) ]
( p r i n t l n ”My −Product ( 1 ∗ 2 ∗ 3 ) : ” my−p r o d u c t ) )


Clearly, this cannot be used for pasting it into a source file or the REPL. The most obvious problem is the spacing of characters but there are also issues related to quotation marks etc.

My current Solution

In order to fix these copy&paste issues, I currently use the following solution:

\documentclass{scrartcl}

\usepackage{listings}
\usepackage[T1]{fontenc}
\usepackage{textcomp}
\usepackage[variablett]{lmodern}

\lstset{
basicstyle=\ttfamily,
columns=fullflexible,
upquote,
keepspaces,
literate={*}{{\char42}}1
{-}{{\char45}}1
}

\begin{document}

\begin{lstlisting}
(let [my-list '(1 2 3)
my-product (reduce * my-list)]
(println "My-Product (1*2*3): " my-product))
\end{lstlisting}

\end{document}


This produces the following output:

Copying and pasting this from the PDF, results in the following:

(let [my-list '(1 2 3)
my-product (reduce * my-list)]
(println "My-Product (1*2*3): " my-product))


Besides the indentation not being correct, this is actually something that can be run as-is in a REPL or be used in a source code file.

The credit for the solution goes to all the helpful posts on this topic. As I had to merge various solution approaches from multiple posts and play a bit with all the input before coming to the above “solution”, I, unfortunately, did not keep track of the references that led me to this state. So I just go with a general “thank you” to the nice and helpful LaTeX community.

## Using the Clojure-based bowerick MoM/JMS Library in Java

First of all, for those who did not run across bowerick yet, here is a brief description: bowerick is a Clojure library that bundles the functionality of various Message-oriented Middleware (MoM)/Java Message Service (JMS) implementations and provides one centralized abstraction for these. It currently supports Stomp, OpenWire, MQTT, and Stomp via WebSockets. You can find more information in other posts I wrote about bowerick.

In this post, I briefly discuss how bowerick, which is written almost entirely in Clojure, can be used in Java. In a sense, this completes a full circle as most (all? But I did not double-check that.) of the MoM/JMS libraries employed in bowerick are Java-based.

My intention is to keep the post focused and concise and to present the most interesting things straight away. Hence, the outline is a bit unusual compared to my other posts:

• bowerick and Java
In this part, I present some Java code snippets etc. for using bowerick in Java.
• bowerick Design Considerations for Java Interoprability
Here, I write about some measures I took in order to allow/ease Java interoperability.
• Closing Remarks

bowerick in Java

In order to demonstrate the use of bowerick in Java, I created a simple example project on github. The example project was set up as Maven project in which the examples were implemented as JUnit test cases. In this section, I will present some snippets of this project.

All transports (Stomp, OpenWire, etc.) supported by bowerick in Clojure can also be used in Java. However, as additional measures have to be taken to make Clojure code available in Java, not all Clojure functions are exposed to Java. Still, I think that the most important features are available in Java as well. The bowerick features currently available in Java are:

• Starting and stopping an embedded JMS broker
• Creating a producer and sending messages
• Creating a consumer and receiving messages
• Selected Serialization Options:
Default, JSON, Kryo (via Carbonite) with Optional LZF Compression

The central point for using bowerick in Java is the JmsController class. E.g., starting and stopping of an embedded broker can be done via an instance of this class as shown in the following listing:

jmsController = new JmsController("stomp://some-url:port");
jmsController.startEmbeddedBroker();
...
jmsController.stopEmbeddedBroker();


This example shows the start/stop of a broker with a single Stomp transport. In addition, a broker with multiple transports can be created by passing a List of Strings to the JmsController constructor as shown in the following listing:

List<String> urls = Arrays.asList(new String[] {
"stomp://127.0.0.1:1701",
"tcp://127.0.0.1:1864",
"ws://127.0.0.1:8472",
"mqtt://127.0.0.1:2000"});
jmsController = new JmsController(urls);
jmsController.startEmbeddedBroker();
...
jmsController.stopEmbeddedBroker();


For the creation of consumers and producers, static methods in JmsController are used. In the following, a simple producer using the example of sending a String is shown:

JmsProducer producer = JmsController.createJsonProducer(
"stomp://127.0.0.1:1701",
"/topic/test, 1);
producer.sendData("A test string");
producer.close();


Similarly, a consumer for receiving data can be created via a static method in JmsController. The following listing shows the creation and usage of a consumer:

final CountDownLatch cdl = new CountDownLatch(1);
final StringBuffer received = new StringBuffer();

AutoCloseable consumer = JmsController.createJsonConsumer(
"stomp://127.0.0.1:1701,
"/topic/test",
new JmsConsumerCallback() {
public void processData(Object data) {
cdl.countDown();
}
},
1);

cdl.await();

consumer.close();


Note that producers and consumers have to be closed when they are not needed anymore. Producers and consumers implement AutoClosable such that they can also be used, e.g., in corresponding try-catch blocks for auto closing the created instances. Furthermore, the consumer example includes a bit more surrounding code in order to demonstrate the use of the received data.

Instead of Strings, more complex data structures, such as Java lists, maps, or sets, can also be sent via these mechanisms.

Regarding the Maven pom.xml, the parts that have to be added are pretty simple as well:

...
<dependencies>
<dependency>
<groupId>bowerick</groupId>
<artifactId>bowerick</artifactId>
<version>1.99.12</version>
</dependency>
</dependencies>
...
<repositories>
<repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>
</repositories>
...


bowerick Design Considerations for Java Interoperability

Even though I primarily use bowerick in Clojure, I also tried to implement it already with Java interoperability in mind. In this section, I will outline the aspects that I consider important in this context.

Clojure offers, in my opinion, excellent Java interoperability such that it is very easy to write Clojure projects that can also be used from within Java. However, in scope of bowerick, I came across some things that I found helpful.

In bowerick, consumers and producers are not just pure functions. In fact, a producer, e.g., has to provide functionality for sending data and functionality for closing and freeing its resources, namely the JMS connection. Hence, there has to be some way for dispatching between these two functions. For bowerick, I chose to define producers and consumers as records and to pass the send and close functions as constructor arguments to these records.

In order to ease the use of these records in Java, I defined the records to implement the Java AutoClosable interface. This is shown for the consumer in the following listing:

(defrecord ConsumerWrapper [close-fn]
AutoCloseable
(close [this]
(close-fn)))


From the perspective of Clojure, just passing a function to the consumer as callback for processing the data would have been sufficient. However, in order to ease the interoperability with Java, I chose to specifically define an interface for this:

(gen-interface
:name bowerick.JmsConsumerCallback
:methods [[processData [Object] void]])


In order to prepare the producer functionality to be available in Java as well, I defined another interface for sending the producer data, in addition to AutoClosable. In the following listing, both the interface definition and the record definition are shown in one listing even though they are defined in separate namespaces in bowerick:

(gen-interface
:name bowerick.JmsProducer
:extends [AutoCloseable]
:methods [[sendData [Object] void]])
...
(defrecord ProducerWrapper [send-fn close-fn]
AutoCloseable
(close [this]
(close-fn))
JmsProducer
(sendData [this data]
(send-fn data))
IFn
(invoke [this data]
(send-fn data)))


Closing Remarks

Personally, I think that this can be an example for a symbiotic co-existence of Java and Clojure. My personal impression is that using Clojure helped me a lot during the development of bowerick and that making bowerick available in Java can benefit on the Java side as well. However, this is just my very subjective impression without any real hard figures to “prove” it.

Regarding the current Java interoperability implementation: I use Object as type in some places, such as the JmsController constructor or the JmsConsumerCallback processData method. For the JmsController, currently supported argument types are, String or List. For the processData method, the type strongly depends on the data that was sent and how it got de-serialized. However, Object being somewhat the NULL pointer equivalent of Java, using just Object breaks the benefits of static typing and requires casting etc., which may be considered disadvantageous.

I hope that this is useful for some. Feedback, comments, and constructive criticism, are always appreciated.

## Unit Testing Arbitrary Interactive Clojure Command Line Interfaces (CLIs) with cli4clj – Illustrated using the Example of the Clojure REPL

In my previous posts, I primarily focused on cli4clj itself and on CLIs created with cli4clj. In this post, I want to address a bit the applicability of the cli4clj unit testing functionality for testing “third-party” CLIs. I will do this by showing how to easily unit test the Clojure REPL with the cli4clj unit testing functionality.

As examples for illustrating the working principles in this post, I will use the source code of unit tests that I recently added to the cli4clj repository. The examples will start simple and will be enhanced step by step in order to provide a hopefully fluent and clear line of argumentation.

In the examples, the input commands will be stored in the “in-cmds” vector, which represents the sequence of inputs that are consecutively entered into the REPL. The output is stored in the “out” var. The expected output that will be emitted is shown as the expected string value in the “(test/is (= “expected string” out))” expressions.

At first, I consider a simple execution of the Clojure REPL with cli4clj. This can be considered as a sort of “Hello World” execution example and is shown in the listing below:

(test/deftest clojure-repl-stdout-no-op-test
(let [in-cmds [""]
out (cli-tests/test-cli-stdout
clojure.main/repl
in-cmds)]
(test/is
(= "user=> user=>" out))))


This example illustrates the basic capability of executing the Clojure REPL with cli4clj. One annoying property is that the REPL prompt output “user=> user=>” will clutter up the output. The problem of the REPL prompt cluttering up the output will get even worse when the complexity of the considered scenarios increases.

In the next step, the problem of the REPL prompt showing up in the output will be tackled. For getting rid of the REPL prompt, the function used for printing the prompt can be set to emit an empty string. In the following listing, the “str” function is used, which, when called without arguments, returns the empty string, in order to remove the REPL prompt:

(test/deftest clojure-repl-stdout-no-op-no-prompt-test
(let [in-cmds [""]
out (cli-tests/test-cli-stdout
#(clojure.main/repl :prompt str)
in-cmds)]
(test/is (= "" out))))


With the REPL prompt output out of the way, it is time to actually do something. In the next example, the “inc” function is used as a kind of “Hello World” for actually executing some functionality:

(test/deftest clojure-repl-stdout-inc-no-prompt-test
(let [in-cmds ["(inc 1)"]
out (cli-tests/test-cli-stdout
#(clojure.main/repl :prompt str)
in-cmds)]
(test/is (= "2" out))))


In the next step, I show a slightly more complex example for actually doing things with the REPL. This example includes the definition of a var, the use of the var, and the execution of a function that prints to stdout:

(test/deftest clojure-repl-stdout-def-inc-println-no-prompt-test
(let [in-cmds ["(def x 21)"
"(inc x)"
"(println x)"]
out (cli-tests/test-cli-stdout
#(clojure.main/repl :prompt str)
in-cmds)]
(test/is (=
(cli-tests/expected-string
["#'user/x"
"22"
"21"
"nil"])
out))))


Instead of the expected result string, as used in the earlier examples, this example uses “cli-tests/expected-string” to build that string. “cli-tests/expected-string” simply takes care of inserting the proper line separator in between the strings given to it in the argument vector.

The example demonstrates that the REPL works as expected. A var can be defined, used, and println outputs the string an returns nil as expected.

However, there is one peculiarity that may not be obvious at first but may become critical later. In this example execution, the var “x” is defined in the “user” namespace and the namespace is hard-coded in the expected string. Under certain circumstances, e.g., when running the tests with “lein cloverage”, however, the namespace in which “x” will be declared will be different and thus the test will fail.

Luckily, this problem can be easily fixed by using the “*ns*” global variable to get the current namespace. This is shown in the following listing:

(test/deftest clojure-repl-stdout-def-inc-println-no-prompt-test
(let [in-cmds ["(def x 21)"
"(inc x)"
"(println x)"]
out (cli-tests/test-cli-stdout
#(clojure.main/repl :prompt str)
in-cmds)]
(test/is (=
(cli-tests/expected-string
[(str "#'" *ns* "/x")
"22"
"21"
"nil"])
out))))


In this example, the hard-coded string “#’user/x” was replaced with a dynamically constructed string based on “*ns*”: “(str “#'” *ns* “/x”)”. I tested this version with “lein test” and “lein cloverage”.

Last but not least, I also show the use of the recently added string-latch. The string-latch can be particularly useful for “stepping” through the execution of a test.

I will not cover details of the string-latch here. For the discussion here, it is sufficient to say that the string latch “sl” waits for the defined strings in a step-wise way. Once one string was matched, it waits for the next string to occur and so on. Thereby, an optional function can be defined that will be executed when the corresponding string matches. An example for this is shown in the listing below:

(test/deftest clojure-repl-stdout-def-inc-println-no-prompt-string-latch-test
(let [in-cmds ["(def x 21)"
"(inc x)"
"(println x)"]
val-0 (atom nil)
val-1 (atom nil)
val-2 (atom nil)
sl (cli-tests/string-latch
[[(str "#'" *ns* "/x") #(reset! val-0 %)]
["22" #(reset! val-1 %)]
["21" #(reset! val-2 %)]
"nil"])
out (cli-tests/test-cli-stdout
#(clojure.main/repl :prompt str)
in-cmds
sl)]
(test/is (= [(str "#'" *ns* "/x")] @val-0))
(test/is (=
[(str "#'" *ns* "/x")
cli/*line-sep*
"22"]
@val-1))
(test/is (=
[(str "#'" *ns* "/x")
cli/*line-sep*
"22"
cli/*line-sep*
"21"]
@val-2))
(test/is (=
(cli-tests/expected-string
[(str "#'" *ns* "/x")
"22"
"21"
"nil"])
out))))


In this example, the string latch is used to set the atom vars “val-0” to 2. The vars are set to the vector of strings that were so far observed by the string-latch “sl”. It can be seen how the strings accumulate with increasing val-x index.

Please note that the namespace was not hard-coded but that the approach using *ns* was used. Furthermore, note that the string latch also contains the new-line characters, which are matched via the cli/*line-sep* placeholder.

Summary

In this post, I used the Clojure REPL as example to show that the unit testing functionality provided by cli4clj can also be used for testing more interactive CLI applications than ones created with cli4clj. So far, I did not encounter limitations such that from my current point of view, it should be possible to test any interactive Clojure CLI application with cli4clj.

I hope that you consider this post and cli4clj useful. As usual, constructive feedback, criticism, and comments are always welcome.

## cli4clj 1.2.5 – Improved Testability of Multi-threaded Command Line Applications in Clojure

In my previous post, I already outlined the problem of testing multi-threaded interactive command line interfaces (CLIs). In the meantime, I came up with some improvements for addressing the testability of multi-threaded command line applications with cli4clj.

In this post I will cover:

• a recap of the problem of testing interactive CLIs vs. multi-threading,
• a first improved solution by synchronizing the testing via CLI output with string-latch,
• and an enhanced string-latch with callback functions.

As long as the functionality associated with a command in a CLI is executed in the main CLI UI thread, testing a CLI is straight forward. One can simply define the commands that are to be executed during a test and, as long all functionality is executed in the main CLI UI thread, the CLI output will be in the order in which the commands are defined. Furthermore, all commands will be completely executed before the CLI UI main thread and with this the test function call terminates. An example of such a test is shown in the following listing:

(test/deftest advanced-main-method-example-test
"divide 4 2"
"d 3 2"]
out-string (cli-tests/test-cli-stdout
#(example/-main "")
test-cmd-input)]
(test/is (=
(cli-tests/expected-string ["3" "2" "3/2"])
out-string))))


However, when the functionality associated with a command is executed in another thread, testing becomes more complicated. In the following listing a contrived example is shown:

(defn- async-test-fn
[]
(println "Starting...")
(let [tmp-out *out*]
(doto
(fn []
(binding [*out* tmp-out]
(utils/sleep 500)
(println "Finished."))))
(.start)))
(println "Started."))

(test/deftest async-cmd-not-finished-test
(let [cli-opts {:cmds {:async-foo {:fn async-test-fn}}}
test-cmd-input ["async-foo"]
out-string (cli-tests/test-cli-stdout
#(cli/start-cli cli-opts)
test-cmd-input)]
(test/is (=
(cli-tests/expected-string
["Starting..." "Started." "Finished."])
out-string))))


The async-cmd-not-finished-test will typically fail. The reason for this is that the execution of async-test-fn spawns a thread in which the “actual processing” (mimicked via utils/sleep) is done. The main CLI UI thread, however, is not aware of this thread and thus only blocks until the test function has terminated.

Consequently, cli-tests/test-cli-stdout will typically terminate before the processing thread has finished. Thus, the output of “Finished.” will typically be not included in out-string such that the test will fail. Even worse, this can result in a race condition with seemingly random behaviour in which tests pass sometimes and sometimes not.

The stop-gap solution introduced back then was to use a hidden _sleep command to cause the main CLI UI thread to wait until the processing thread had finished. However, using sleeps for this kind of use case is not a good solution.

Synchronized Testing via CLI Output with string-latch

In order to improve the testability of multi-threaded CLIs, the execution of non-UI threads has to be synchronized with the test-execution of the main CLI UI thread. As this is about CLIs, one natural option for this synchronization is to use the CLI output for synchronization. In the end, even the execution of multi-threaded functionality in an interactive CLI can be typically expected to cause some sort of CLI output.

For synchronizing the test execution based on the CLI output, I added what I call “string-latch”. The name is inspired by the Java CountDownLatch, which is used under the hood. In its simplest form, a string-latch is defined as a vector of strings that are expected to occur in the given order. Thereby, it is sufficient to only define key elements of the expected output such that it is not required to exactly reproduce the actual test CLI output. In the following listing, the string-latch solution for the above example problem async-cmd-not-finished-test is given:

(test/deftest async-cmd-string-latch-stdout-test
(let [cli-opts {:cmds {:async-foo {:fn async-test-fn}}}
test-cmd-input ["async-foo"]
sl (cli-tests/string-latch ["Finished." "\n"])
out-string (cli-tests/test-cli-stdout
#(cli/start-cli cli-opts)
test-cmd-input
sl)]
(test/is (=
["Starting..." "\n" "Started." "\n" "Finished." "\n"]
(sl)))
(test/is (=
(cli-tests/expected-string
["Starting..." "Started." "Finished."])
out-string))))


In comparison to the problematic test async-cmd-not-finished-test, the changes are: the addition of the string-latch “sl” and the use of the test-cli-stdout that takes the string-latch as argument.

String Latch with Callback Functions

As another improvement, I added the possibility to execute callback functions for string-latch entries. These callback functions will be executed when the defined string-latch entry was matched. In order to define string-latch entries with callbacks, a vector containing the match-string and the callback is used as string-latch entry instead of a simple string. The following listing shows an example that builds up on the previous examples:

(test/deftest async-cmd-string-latch-stdout-with-callback-test
(let [cli-opts {:cmds {:async-foo {:fn async-test-fn}}}
test-cmd-input ["async-foo"]
val-0 (atom nil)
val-1 (atom nil)
val-2 (atom nil)
sl (cli-tests/string-latch
[["Starting..." #(reset! val-0 %)]
["Started." (fn [v] (reset! val-1 v))]
["Finished." #(reset! val-2 %)]
"\n"])
out-string (cli-tests/test-cli-stdout
#(cli/start-cli cli-opts)
test-cmd-input sl)]
(test/is (= ["Starting..."] @val-0))
(test/is (= ["Starting..." "\n" "Started."] @val-1))
(test/is (=
["Starting..." "\n" "Started." "\n" "Finished."]
@val-2))
(test/is (=
(cli-tests/expected-string
["Starting..." "Started." "Finished."])
out-string))))


The string and the extended, vector with callback, versions of string-latch entries can also be mixed. This is shown in the following listing:

(test/deftest async-cmd-string-latch-stdout-with-mixed-callback-test
(let [cli-opts {:cmds {:async-foo {:fn async-test-fn}}}
test-cmd-input ["async-foo"]
val-0 (atom nil)
val-1 (atom nil)
val-2 (atom nil)
sl (cli-tests/string-latch ["Starting..."
["Started."]
["Finished." #(reset! val-2 %)]
"\n"])
out-string (cli-tests/test-cli-stdout
#(cli/start-cli cli-opts)
test-cmd-input
sl)]
(test/is (= nil @val-0))
(test/is (= nil @val-1))
(test/is (=
["Starting..." "\n" "Started." "\n" "Finished."]
@val-2))
(test/is (=
(cli-tests/expected-string
["Starting..." "Started." "Finished."])
out-string))))


Summary

In this post, I briefly introduced the string-latch for testing interactive CLIs and its most important features. I think the string-latch is an important improvement in comparison to the previous _sleep stop-gap solution. For my use cases, the string-latch works very well so far. I hope to get the time to write a bit more about the details of the string-latch in future posts.

As usual, you can get the latest version of cli4clj from clojars.org and I would be happy if you left some constructive feedback or comments.

## cli4clj Version 1.2.3 Released

cli4clj is a library for easing the implementation of simple interactive command line interfaces (CLIs) for Clojure applications. In this post, I briefly announce the release of cli4clj version 1.2.3.

The most important changes in version 1.2.3 are:

• Add support for hiding commands in help and completion.
Hidden commands are prefixed with “_”.
• Hide the “enable-trace” command by changing it to “_enable-trace”.
• Add a hidden “_sleep” command.

For hiding commands, the convention of prefixing hidden commands with “_” was introduced. This means that all commands that begin with an underscore, “_”, will neither be shown in the help nor in the command completion.

The rationale for hiding certain commands is to improve the usability by removing everything a “typical” user does not need. In this process, I also changed the “enable-trace” command to “_enable-trace” in order to hide it from the user.

The “_sleep” command was added as a stopgap solution for testing multi-threaded CLI applications. The current CLI test functionality works very well. However, there is a problem with multi-threaded applications.

Clearly, when a command triggers interaction with another thread that eventually prints a result, a corresponding unit test needs to wait until the processing finished and the result was printed. The CLI “UI thread” however, cannot know how long it has to wait. Thus, I added the “_sleep” command as a simple stopgap solution.

In the following, the code of the newly added unit tests for testing the “_sleep” command is shown for illustrating the problem and the current stopgap “solution”:

(deftest async-cmd-not-finished-test
(let [cli-opts {:cmds {:async-foo {:fn (fn []
(println "Starting...")
(let [tmp-out *out*]
(doto
(fn []
(binding [*out* tmp-out]
(sleep 500)
(println "Finished."))))
(.start)))
"Started.")}}}
test-cmd-input ["async-foo"]
out-string (test-cli-stdout #(start-cli cli-opts) test-cmd-input)]
(is (= (expected-string ["Starting..." "\"Started.\""]) out-string))))

(deftest async-cmd-sleep-finished-test
(let [cli-opts {:cmds {:async-foo {:fn (fn []
(println "Starting...")
(let [tmp-out *out*]
(doto
(fn []
(binding [*out* tmp-out]
(sleep 500)
(println "Finished."))))
(.start)))
"Started.")}}}
test-cmd-input ["async-foo" "_sleep 1000"]
out-string (test-cli-stdout #(start-cli cli-opts) test-cmd-input)]
(is (= (expected-string ["Starting..." "\"Started.\"" "Finished."]) out-string))))


Actually, using sleeps for this is not really an ideal solution. However, for now, this approach serves its purpose. In future, I may come up with something better that still allows me to keep cli4clj simple.

As usual, constructive criticism, feedback, and comments are always appreciated.

Posted in cli4clj, Libs. | Tagged , , | 1 Comment

## Playing a bit with Message-oriented Middleware Protocols/Transports using bowerick

bowerick combines various libraries, e.g., for Message-oriented Middleware (MoM) protocols/transports. One aim was to provide an easily usable abstraction for playing and experimenting with MoM implementations. In this post, I use bowerick for a simple toy experiment to illustrate some of the abstractions provided by bowerick and some MoM protocol/transport related aspects.

The toy example discussed in this post is: forwarding of messages to multiple MoM protocols/transports via a single common broker instance. The following protocols were used:

bowerick uses ActiveMQ as embedded broker and for OpenWire TCP and UDP clients. In addition, bowerick uses further libraries for providing easily usable STOMP, WebSocket, and MQTT clients. For a full list of libraries please refer to bowericks dependencies, which can be found, e.g., in bowericks project.clj file.

The outline of this post is as follows:

• Toy Example and Code
At first, the toy example is introduced and the example code is explained.
• Raw Network Packet Data
Subsequently, raw network packet data for the packets transporting the payload is shown.
• TCP Streams
Afterwards, the TCP streams that carried the data are shown.
• Concluding Remarks
Finally, this post is finished with some concluding remarks.

Toy Example and Code

The ActiveMQ broker already supports forwarding of messages between protocols/transports. bowerick adds a bit of convenience to this in the way that it eases the way how a broker that services multiple protocols/transports can be started. In the following listing, the first part of the toy example for this post, which can, e.g., be run in a REPL, is shown:

(def urls ["tcp://127.0.0.1:55511" "stomp://127.0.0.1:55522" "ws://127.0.0.1:55533" "mqtt://127.0.0.1:55544" "udp://127.0.0.1:55555"])
(def broker (bowerick.jms/start-broker urls))


With bowerick, multiple transports are simply defined as a vector of strings (URLs) in which each string defines one transport via the URL format that is typically used by ActiveMQ. The embedded broker is simply started via the bowerick.jms/start-broker function which takes this URLs vector as argument and returns a reference to the started embedded broker instance.

In the next step, consumers are connected via the aforementioned transports (URLs) to the same test topic. As all consumers connect to the same topic, the ActiveMQ broker will forward messages sent to this topic via these transports. Furthermore, a single producer is created that is used for sending a map to the test topic via the OpenWire TCP transport. This is shown in the following listing:

(def test-topic "/topic/testtopic.foo")
(def consumers (doall (map-indexed (fn [idx url] (bowerick.jms/create-json-consumer url test-topic (fn [rcvd] (clj-assorted-utils.util/sleep (* 50 idx)) (println (aget (.split url ":") 0) "->" rcvd)))) urls)))
(def prod (bowerick.jms/create-json-producer (first urls) test-topic))
(prod {"a" 123 "b" "xyz"})


The somewhat lengthy definition of the consumers is simply used as convenience for creating consumers for all transport URLs in the urls vector. For each successfully received message, the consumers print a prefix corresponding to the URL and the received data. The print output is artificially delayed by sleeping depending on the index of the URL string in the vector in order to enforce that the order of the printed output corresponds to the order of the URLs in the vector.

The data is transmitted as byte array of the datas JSON UTF-8 string representation. In the following listing, the output of the consumers is shown:

...
tcp -> {a 123, b xyz}
stomp -> {a 123, b xyz}
ws -> {a 123, b xyz}
mqtt -> {a 123, b xyz}
udp -> {a 123, b xyz}
...


For the sake of completeness, the following listing shows the shutdown procedure for the producer, consumers, and broker:

(bowerick.jms/close prod)
(doseq [c consumers] (bowerick.jms/close c))
(bowerick.jms/stop broker)


Raw Packet Data

During the experiment, I captured the network traffic with Wireshark. Besides the fun aspects of this, my intention was also to verify and showcase that bowerick indeed uses the advertised protocols. In the end, having some program output claim the used protocol/transport was, e.g., STOMP is not as trustworthy as actually seeing it on “the wire”.

In the following, I first print the captured packets that carried the actual payload. As the payload was serialized to the byte array representation of a UTF-8 encoded JSON string, the payload can be quite easily identified in this data. Please note that these are only the packets with the actual payload and that there further packets were exchanged throughout this experiment.

Just in case, if you are note familiar with the structure of this type of packet dump: the first column shows the offset of the packet data in bytes as hexadecimal (hex) value. The second column shows the raw packet data in hex in groups of 1 byte. The third column shows the string representation of the raw packet data. As a little bit more aid for reading the packet data, the loopback address, 127.0.0.1, in hex is “7f 00 00 01”, and the relevant port number that were used are: 55511 (“d8 d7”), 55522 (“d8 e2”), 55533 (“d8 ed”), 55544 (“d8 f8”), 55555 (“d9 03”).

Producer

At first the packet with the payload that was sent from the producer is shown:

Producer: OpenWire (TCP) (“tcp://127.0.0.1:55511”)

0000    02 00 00 00 45 00 00 7c ee 41 40 00 40 06 00 00    ....E..|.A@.@...
0010    7f 00 00 01 7f 00 00 01 a9 78 d8 d7 ea 2e 7d da    .........x....}.
0020    61 77 90 b1 80 18 04 f3 fe 70 00 00 01 01 08 0a    aw.......p......
0030    0c e4 e0 bf 50 ea e1 a6 00 00 00 44 18 05 48 04    ....P......D..H.
0040    b0 00 00 00 00 00 05 00 04 00 05 00 06 00 06 6e    ...............n
0050    00 04 00 01 00 06 00 00 00 00 04 00 00 01 56 fe    ..............V.
0060    6b 5a e0 00 00 00 13 7b 22 61 22 3a 31 32 33 2c    kZ.....{"a":123,
0070    22 62 22 3a 22 78 79 7a 22 7d 00 06 00 00 00 00    "b":"xyz"}......


Consumers

Next, the packets that contained the payload that were sent to the consumers are shown:

Consumer: OpenWire (TCP) (“tcp://127.0.0.1:55511”)

0000    02 00 00 00 45 00 00 fd ee 43 40 00 40 06 00 00    ....E....C@.@...
0010    7f 00 00 01 7f 00 00 01 d8 d7 72 58 37 d6 32 db    ..........rX7.2.
0020    c6 20 11 15 80 18 04 00 fe f1 00 00 01 01 08 0a    . ..............
0030    52 68 2f 13 0c e4 bf 97 00 00 00 c5 15 08 5e 3f    Rh/...........^?
0040    af 12 05 2c 00 1e 00 00 00 00 00 01 7a 00 20 49    ...,........z. I
0050    44 3a 63 6f 6c 69 6e 2d 33 37 39 39 37 2d 31 34    D:colin-37997-14
0060    37 33 31 34 37 32 35 32 33 36 33 2d 38 3a 31 00    73147252363-8:1.
0070    01 00 01 00 02 65 00 0d 74 65 73 74 74 6f 70 69    .....e..testtopi
0080    63 2e 66 6f 6f 18 00 00 00 05 00 03 7b 00 21 49    c.foo.......{.!I
0090    44 3a 63 6f 6c 69 6e 2d 33 37 39 39 37 2d 31 34    D:colin-37997-14
00a0    37 33 31 34 37 32 35 32 33 36 33 2d 31 36 3a 31    73147252363-16:1
00b0    00 01 00 01 00 02 00 04 00 04 6e 00 03 00 01 00    ..........n.....
00c0    15 00 04 00 00 00 00 04 00 00 01 56 fe 6b 5a e0    ...........V.kZ.
00d0    00 00 00 13 7b 22 61 22 3a 31 32 33 2c 22 62 22    ....{"a":123,"b"
00e0    3a 22 78 79 7a 22 7d 00 04 00 00 00 00 00 00 01    :"xyz"}.........
00f0    56 fe 6b 5a e3 00 00 01 56 fe 6b 5a eb 00 00 00    V.kZ....V.kZ....
0100    00


Consumer: STOMP (TCP) (“stomp://127.0.0.1:55522”)

0000    02 00 00 00 45 00 00 f9 ee 46 40 00 40 06 00 00    ....E....F@.@...
0010    7f 00 00 01 7f 00 00 01 d8 e2 f0 29 f4 ea d6 84    ...........)....
0020    da 4f ae 7e 80 18 04 00 fe ed 00 00 01 01 08 0a    .O.~............
0030    7d e0 1b ae 0c e4 c1 7a 4d 45 53 53 41 47 45 0a    }......zMESSAGE.
0040    63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 31    content-length:1
0050    39 0a 65 78 70 69 72 65 73 3a 30 0a 64 65 73 74    9.expires:0.dest
0060    69 6e 61 74 69 6f 6e 3a 2f 74 6f 70 69 63 2f 74    ination:/topic/t
0070    65 73 74 74 6f 70 69 63 2e 66 6f 6f 0a 73 75 62    esttopic.foo.sub
0080    73 63 72 69 70 74 69 6f 6e 3a 31 0a 70 72 69 6f    scription:1.prio
0090    72 69 74 79 3a 34 0a 6d 65 73 73 61 67 65 2d 69    rity:4.message-i
00a0    64 3a 49 44 5c 63 63 6f 6c 69 6e 2d 33 37 39 39    d:ID\ccolin-3799
00b0    37 2d 31 34 37 33 31 34 37 32 35 32 33 36 33 2d    7-1473147252363-
00c0    31 36 5c 63 31 5c 63 31 5c 63 31 5c 63 31 0a 74    16\c1\c1\c1\c1.t
00d0    69 6d 65 73 74 61 6d 70 3a 31 34 37 33 31 34 37    imestamp:1473147
00e0    32 36 33 37 31 32 0a 0a 7b 22 61 22 3a 31 32 33    263712..{"a":123
00f0    2c 22 62 22 3a 22 78 79 7a 22 7d 00 0a             ,"b":"xyz"}..


Consumer: WebSocket (TCP) (“ws://127.0.0.1:55533”)

0000    02 00 00 00 45 00 00 f7 ee 47 40 00 40 06 00 00    ....E....G@.@...
0010    7f 00 00 01 7f 00 00 01 d8 ed 2b 82 b2 30 0b 74    ..........+..0.t
0020    9a c7 80 6c 80 18 04 fb fe eb 00 00 01 01 08 0a    ...l............
0030    6d 7b db 92 0c e4 c1 d2 81 7e 00 bf 4d 45 53 53    m{.......~..MESS
0040    41 47 45 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67    AGE.content-leng
0050    74 68 3a 31 39 0a 65 78 70 69 72 65 73 3a 30 0a    th:19.expires:0.
0060    64 65 73 74 69 6e 61 74 69 6f 6e 3a 2f 74 6f 70    destination:/top
0070    69 63 2f 74 65 73 74 74 6f 70 69 63 2e 66 6f 6f    ic/testtopic.foo
0080    0a 73 75 62 73 63 72 69 70 74 69 6f 6e 3a 30 0a    .subscription:0.
0090    70 72 69 6f 72 69 74 79 3a 34 0a 6d 65 73 73 61    priority:4.messa
00a0    67 65 2d 69 64 3a 49 44 3a 63 6f 6c 69 6e 2d 33    ge-id:ID:colin-3
00b0    37 39 39 37 2d 31 34 37 33 31 34 37 32 35 32 33    7997-14731472523
00c0    36 33 2d 31 36 3a 31 3a 31 3a 31 3a 31 0a 74 69    63-16:1:1:1:1.ti
00d0    6d 65 73 74 61 6d 70 3a 31 34 37 33 31 34 37 32    mestamp:14731472
00e0    36 33 37 31 32 0a 0a 7b 22 61 22 3a 31 32 33 2c    63712..{"a":123,
00f0    22 62 22 3a 22 78 79 7a 22 7d 00                   "b":"xyz"}.


Note that the packets for STOMP and WebSocket show quite strong similarities. The reason for this is that the WebSocket transport used in bowerick uses STOMP as MoM protocol. However, there are also differences due to the differences between the two transports.

Consumer: MQTT (TCP) (“mqtt://127.0.0.1:55544”)

0000    02 00 00 00 45 00 00 5a ee 45 40 00 40 06 00 00    ....E..Z.E@.@...
0010    7f 00 00 01 7f 00 00 01 d8 f8 5f bd c8 97 d2 49    .........._....I
0020    2b 9d 6d 9c 80 18 04 00 fe 4e 00 00 01 01 08 0a    +.m......N......
0030    63 7e 77 da 0c e4 c2 85 32 24 00 0d 74 65 73 74    c~w.....2$..test 0040 74 6f 70 69 63 2f 66 6f 6f 00 01 7b 22 61 22 3a topic/foo..{"a": 0050 31 32 33 2c 22 62 22 3a 22 78 79 7a 22 7d 123,"b":"xyz"}  Consumer: OpenWire (UDP) (“udp://127.0.0.1:55555”) 0000 02 00 00 00 45 00 01 65 ee 44 00 00 40 11 00 00 ....E..e.D..@... 0010 7f 00 00 01 7f 00 00 01 ac 91 3c 67 01 51 ff 64 ..........<g.Q.d 0020 00 00 01 45 15 00 00 00 06 00 01 7a 01 00 21 49 ...E.......z..!I 0030 44 3a 63 6f 6c 69 6e 2d 33 37 39 39 37 2d 31 34 D:colin-37997-14 0040 37 33 31 34 37 32 35 32 33 36 33 2d 31 34 3a 31 73147252363-14:1 0050 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 ................ 0060 01 65 01 00 0d 74 65 73 74 74 6f 70 69 63 2e 66 .e...testtopic.f 0070 6f 6f 01 18 00 00 00 05 00 01 7b 01 00 21 49 44 oo........{..!ID 0080 3a 63 6f 6c 69 6e 2d 33 37 39 39 37 2d 31 34 37 :colin-37997-147 0090 33 31 34 37 32 35 32 33 36 33 2d 31 36 3a 31 00 3147252363-16:1. 00a0 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 01 ................ 00b0 65 01 00 0d 74 65 73 74 74 6f 70 69 63 2e 66 6f e...testtopic.fo 00c0 6f 00 00 01 6e 00 01 7b 01 00 21 49 44 3a 63 6f o...n..{..!ID:co 00d0 6c 69 6e 2d 33 37 39 39 37 2d 31 34 37 33 31 34 lin-37997-147314 00e0 37 32 35 32 33 36 33 2d 31 36 3a 31 00 00 00 00 7252363-16:1.... 00f0 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 ................ 0100 00 00 00 01 00 00 00 00 00 00 00 15 00 00 00 00 ................ 0110 00 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 ................ 0120 01 56 fe 6b 5a e0 00 01 00 00 00 13 7b 22 61 22 .V.kZ.......{"a" 0130 3a 31 32 33 2c 22 62 22 3a 22 78 79 7a 22 7d 00 :123,"b":"xyz"}. 0140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0150 00 00 00 00 00 00 01 56 fe 6b 5a e3 00 00 01 56 .......V.kZ....V 0160 fe 6b 5a f3 00 00 00 00 00 .kZ......  TCP Streams for Consumers In addition to the raw packet data, I also print the TCP streams that contained the payload data for the consumers. Similarly to the packet data, please note that these are only the streams that contained the actual payload and I did not check if additional streams were involved during the communication. However, you can download the pcap file from which this data was taken from the bowerick repository. Consumer: OpenWire (TCP) (“tcp://127.0.0.1:55511”) For the OpenWire TCP stream, I manually wrapped the data either at hopefully meaningful offsets or alternatively at column 80 in order to avoid the lines getting too long. .....ActiveMQ............ ..StackTraceEnabled....PlatformDetails..NJVM: 1.8.0_92, 25.92-b14, Oracle Corpo ration, OS: FreeBSD, 10.3-RELEASE, amd64..CacheEnabled....Host...127.0.0.1..Tcp NoDelayEnabled....SizePrefixDisabled....CacheSize.......ProviderName...ActiveMQ ..TightEncodingEnabled....MaxFrameSize...........MaxInactivityDuration.......u0 . MaxInactivityDurationInitalDelay.......'...ProviderVersion...5.14.0 .....ActiveMQ........s......TcpNoDelayEnabled....SizePrefixDisabled....CacheSiz e.......ProviderName...ActiveMQ..StackTraceEnabled....PlatformDetails..NJVM: 1. 8.0_92, 25.92-b14, Oracle Corporation, OS: FreeBSD, 10.3-RELEASE, amd64..CacheE nabled....TightEncodingEnabled....MaxFrameSize...........MaxInactivityDuration. ......u0. MaxInactivityDurationInitalDelay.......'...ProviderVersion...5.14.0.. .P..~........|. ID:colin-37997-1473147252363-0:1..tcp://localhost:55511..localh ost ...O..........x. ID:colin-37997-1473147252363-8:1. ID:colin-37997-1473147252363 -7:1..................................~...}........z. ID:colin-37997-1473147252 363-8:1............e.7ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic.. ............................^......y. ID:colin-37997-1473147252363-8:1.....p.._ }........z. ID:colin-37997-1473147252363-8:1......e. testtopic.foo......... ID:colin-37997-1473147252363-7:1. .....................^?...,........z. ID:colin-37997-1473147252363-8:1......e. testtopic.foo.......{.!ID:colin-37997-1473147252363-16:1..........n............ ....V.kZ.....{"a":123,"b":"xyz"}.........V.kZ....V.kZ........K...W............. n..{.!ID:colin-37997-1473147252363-16:1........n..........  Consumer: STOMP (TCP) (“stomp://127.0.0.1:55522”) CONNECT accept-version:1.1 host:127.0.0.1 . CONNECTED server:ActiveMQ/5.14.0 heart-beat:0,0 session:ID:colin-37997-1473147252363-9:1 version:1.1 . SUBSCRIBE receipt:2 ack:client destination:/topic/testtopic.foo id:1 . RECEIPT receipt-id:2 . MESSAGE content-length:19 expires:0 destination:/topic/testtopic.foo subscription:1 priority:4 message-id:ID\ccolin-37997-1473147252363-16\c1\c1\c1\c1 timestamp:1473147263712 {"a":123,"b":"xyz"}. ACK receipt:3 subscription:1 message-id:ID\ccolin-37997-1473147252363-16\c1\c1\c1\c1 . RECEIPT receipt-id:3 .  Consumer: WebSocket (TCP) (“ws://127.0.0.1:55533”) GET / HTTP/1.1 Host: 127.0.0.1:55533 Upgrade: websocket Connection: Upgrade Sec-WebSocket-Key: QzJg5xeq9ERBZCTZLR7Ugw== Sec-WebSocket-Version: 13 Pragma: no-cache Cache-Control: no-cache HTTP/1.1 101 Switching Protocols Date: Tue, 06 Sep 2016 07:34:15 GMT Connection: Upgrade Sec-WebSocket-Accept: xjAqlX7Q/bGXOz0vro4ef+UXle4= Sec-WebSocket-Protocol: stomp Upgrade: WebSocket ...3...|...p...V.......G.......P.......@...........9...gCONNECTED server:ActiveMQ/5.14.0 heart-beat:0,0 session:ID:colin-37997-1473147252363-9:2 version:1.2 .....n.}.,Um.'Dk. c]..hO..i@.ArA..e...uZ..vG.@A.doJ.^.$..~..MESSAGE
content-length:19
expires:0
destination:/topic/testtopic.foo
subscription:0
priority:4
message-id:ID:colin-37997-1473147252363-16:1:1:1:1
timestamp:1473147263712

{"a":123,"b":"xyz"}.


Consumer: MQTT (TCP) (“mqtt://127.0.0.1:55544”)

....MQTT...<..paho216318442465645 ........
testtopic/foo......2\$.
testtopic/foo..{"a":123,"b":"xyz"}@...
`

Concluding Remarks

With respect to MoM protocols/transports, the ActiveMQ broker is capable of forwarding data to various MoM protocols/transports. bowerick adds further convenience functionality on top of this.

A simple toy example was used to illustrate the use of multiple MoM protocols/transports with bowerick. In addition, raw network packet data as well as TCP stream data of some parts of the corresponding communication were shown.

With respect to the serialization, I used my current favorite serialization approach from an inter-operation perspective: byte arrays of UTF-8 encoded JSON strings. This way of serializing data provides, in my opinion, excellent inter-operation capabilities with respect to, e.g., MoM protocols/transports as well as programming language combinations.

With respect to the MoM protocol/transport, I quite like the simplicity of STOMP. The captured STOMP TCP stream nicely shows the clarity of this protocol. On the other hand, I also liked MQTT because of its minimalism.

I hope that this post is useful. As usual, constructive criticism, comments, and suggestions are highly appreciated.

Posted in bowerick, Libs. | Tagged , , , , , , , , | Leave a comment