bowerick combines various libraries, e.g., for Message-oriented Middleware (MoM) protocols/transports. One aim was to provide an easily usable abstraction for playing and experimenting with MoM implementations. In this post, I use bowerick for a simple toy experiment to illustrate some of the abstractions provided by bowerick and some MoM protocol/transport related aspects.
The toy example discussed in this post is: forwarding of messages to multiple MoM protocols/transports via a single common broker instance. The following protocols were used:
- OpenWire (TCP)
- STOMP (TCP)
- WebSockets (TCP)
- MQTT (TCP)
- OpenWire (UDP)
bowerick uses ActiveMQ as embedded broker and for OpenWire TCP and UDP clients. In addition, bowerick uses further libraries for providing easily usable STOMP, WebSocket, and MQTT clients. For a full list of libraries please refer to bowericks dependencies, which can be found, e.g., in bowericks project.clj file.
The outline of this post is as follows:
- Toy Example and Code
At first, the toy example is introduced and the example code is explained. - Raw Network Packet Data
Subsequently, raw network packet data for the packets transporting the payload is shown. - TCP Streams
Afterwards, the TCP streams that carried the data are shown. - Concluding Remarks
Finally, this post is finished with some concluding remarks.
Toy Example and Code
The ActiveMQ broker already supports forwarding of messages between protocols/transports. bowerick adds a bit of convenience to this in the way that it eases the way how a broker that services multiple protocols/transports can be started. In the following listing, the first part of the toy example for this post, which can, e.g., be run in a REPL, is shown:
(def urls ["tcp://127.0.0.1:55511" "stomp://127.0.0.1:55522" "ws://127.0.0.1:55533" "mqtt://127.0.0.1:55544" "udp://127.0.0.1:55555"]) (def broker (bowerick.jms/start-broker urls))
With bowerick, multiple transports are simply defined as a vector of strings (URLs) in which each string defines one transport via the URL format that is typically used by ActiveMQ. The embedded broker is simply started via the bowerick.jms/start-broker function which takes this URLs vector as argument and returns a reference to the started embedded broker instance.
In the next step, consumers are connected via the aforementioned transports (URLs) to the same test topic. As all consumers connect to the same topic, the ActiveMQ broker will forward messages sent to this topic via these transports. Furthermore, a single producer is created that is used for sending a map to the test topic via the OpenWire TCP transport. This is shown in the following listing:
(def test-topic "/topic/testtopic.foo")
(def consumers (doall (map-indexed (fn [idx url] (bowerick.jms/create-json-consumer url test-topic (fn [rcvd] (clj-assorted-utils.util/sleep (* 50 idx)) (println (aget (.split url ":") 0) "->" rcvd)))) urls)))
(def prod (bowerick.jms/create-json-producer (first urls) test-topic))
(prod {"a" 123 "b" "xyz"})
The somewhat lengthy definition of the consumers is simply used as convenience for creating consumers for all transport URLs in the urls vector. For each successfully received message, the consumers print a prefix corresponding to the URL and the received data. The print output is artificially delayed by sleeping depending on the index of the URL string in the vector in order to enforce that the order of the printed output corresponds to the order of the URLs in the vector.
The data is transmitted as byte array of the datas JSON UTF-8 string representation. In the following listing, the output of the consumers is shown:
...
tcp -> {a 123, b xyz}
stomp -> {a 123, b xyz}
ws -> {a 123, b xyz}
mqtt -> {a 123, b xyz}
udp -> {a 123, b xyz}
...
For the sake of completeness, the following listing shows the shutdown procedure for the producer, consumers, and broker:
(bowerick.jms/close prod) (doseq [c consumers] (bowerick.jms/close c)) (bowerick.jms/stop broker)
Raw Packet Data
During the experiment, I captured the network traffic with Wireshark. Besides the fun aspects of this, my intention was also to verify and showcase that bowerick indeed uses the advertised protocols. In the end, having some program output claim the used protocol/transport was, e.g., STOMP is not as trustworthy as actually seeing it on “the wire”.
In the following, I first print the captured packets that carried the actual payload. As the payload was serialized to the byte array representation of a UTF-8 encoded JSON string, the payload can be quite easily identified in this data. Please note that these are only the packets with the actual payload and that there further packets were exchanged throughout this experiment.
Just in case, if you are note familiar with the structure of this type of packet dump: the first column shows the offset of the packet data in bytes as hexadecimal (hex) value. The second column shows the raw packet data in hex in groups of 1 byte. The third column shows the string representation of the raw packet data. As a little bit more aid for reading the packet data, the loopback address, 127.0.0.1, in hex is “7f 00 00 01”, and the relevant port number that were used are: 55511 (“d8 d7”), 55522 (“d8 e2”), 55533 (“d8 ed”), 55544 (“d8 f8”), 55555 (“d9 03”).
Producer
At first the packet with the payload that was sent from the producer is shown:
Producer: OpenWire (TCP) (“tcp://127.0.0.1:55511”)
0000 02 00 00 00 45 00 00 7c ee 41 40 00 40 06 00 00 ....E..|.A@.@...
0010 7f 00 00 01 7f 00 00 01 a9 78 d8 d7 ea 2e 7d da .........x....}.
0020 61 77 90 b1 80 18 04 f3 fe 70 00 00 01 01 08 0a aw.......p......
0030 0c e4 e0 bf 50 ea e1 a6 00 00 00 44 18 05 48 04 ....P......D..H.
0040 b0 00 00 00 00 00 05 00 04 00 05 00 06 00 06 6e ...............n
0050 00 04 00 01 00 06 00 00 00 00 04 00 00 01 56 fe ..............V.
0060 6b 5a e0 00 00 00 13 7b 22 61 22 3a 31 32 33 2c kZ.....{"a":123,
0070 22 62 22 3a 22 78 79 7a 22 7d 00 06 00 00 00 00 "b":"xyz"}......
Consumers
Next, the packets that contained the payload that were sent to the consumers are shown:
Consumer: OpenWire (TCP) (“tcp://127.0.0.1:55511”)
0000 02 00 00 00 45 00 00 fd ee 43 40 00 40 06 00 00 ....E....C@.@...
0010 7f 00 00 01 7f 00 00 01 d8 d7 72 58 37 d6 32 db ..........rX7.2.
0020 c6 20 11 15 80 18 04 00 fe f1 00 00 01 01 08 0a . ..............
0030 52 68 2f 13 0c e4 bf 97 00 00 00 c5 15 08 5e 3f Rh/...........^?
0040 af 12 05 2c 00 1e 00 00 00 00 00 01 7a 00 20 49 ...,........z. I
0050 44 3a 63 6f 6c 69 6e 2d 33 37 39 39 37 2d 31 34 D:colin-37997-14
0060 37 33 31 34 37 32 35 32 33 36 33 2d 38 3a 31 00 73147252363-8:1.
0070 01 00 01 00 02 65 00 0d 74 65 73 74 74 6f 70 69 .....e..testtopi
0080 63 2e 66 6f 6f 18 00 00 00 05 00 03 7b 00 21 49 c.foo.......{.!I
0090 44 3a 63 6f 6c 69 6e 2d 33 37 39 39 37 2d 31 34 D:colin-37997-14
00a0 37 33 31 34 37 32 35 32 33 36 33 2d 31 36 3a 31 73147252363-16:1
00b0 00 01 00 01 00 02 00 04 00 04 6e 00 03 00 01 00 ..........n.....
00c0 15 00 04 00 00 00 00 04 00 00 01 56 fe 6b 5a e0 ...........V.kZ.
00d0 00 00 00 13 7b 22 61 22 3a 31 32 33 2c 22 62 22 ....{"a":123,"b"
00e0 3a 22 78 79 7a 22 7d 00 04 00 00 00 00 00 00 01 :"xyz"}.........
00f0 56 fe 6b 5a e3 00 00 01 56 fe 6b 5a eb 00 00 00 V.kZ....V.kZ....
0100 00
Consumer: STOMP (TCP) (“stomp://127.0.0.1:55522”)
0000 02 00 00 00 45 00 00 f9 ee 46 40 00 40 06 00 00 ....E....F@.@...
0010 7f 00 00 01 7f 00 00 01 d8 e2 f0 29 f4 ea d6 84 ...........)....
0020 da 4f ae 7e 80 18 04 00 fe ed 00 00 01 01 08 0a .O.~............
0030 7d e0 1b ae 0c e4 c1 7a 4d 45 53 53 41 47 45 0a }......zMESSAGE.
0040 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 31 content-length:1
0050 39 0a 65 78 70 69 72 65 73 3a 30 0a 64 65 73 74 9.expires:0.dest
0060 69 6e 61 74 69 6f 6e 3a 2f 74 6f 70 69 63 2f 74 ination:/topic/t
0070 65 73 74 74 6f 70 69 63 2e 66 6f 6f 0a 73 75 62 esttopic.foo.sub
0080 73 63 72 69 70 74 69 6f 6e 3a 31 0a 70 72 69 6f scription:1.prio
0090 72 69 74 79 3a 34 0a 6d 65 73 73 61 67 65 2d 69 rity:4.message-i
00a0 64 3a 49 44 5c 63 63 6f 6c 69 6e 2d 33 37 39 39 d:ID\ccolin-3799
00b0 37 2d 31 34 37 33 31 34 37 32 35 32 33 36 33 2d 7-1473147252363-
00c0 31 36 5c 63 31 5c 63 31 5c 63 31 5c 63 31 0a 74 16\c1\c1\c1\c1.t
00d0 69 6d 65 73 74 61 6d 70 3a 31 34 37 33 31 34 37 imestamp:1473147
00e0 32 36 33 37 31 32 0a 0a 7b 22 61 22 3a 31 32 33 263712..{"a":123
00f0 2c 22 62 22 3a 22 78 79 7a 22 7d 00 0a ,"b":"xyz"}..
Consumer: WebSocket (TCP) (“ws://127.0.0.1:55533”)
0000 02 00 00 00 45 00 00 f7 ee 47 40 00 40 06 00 00 ....E....G@.@...
0010 7f 00 00 01 7f 00 00 01 d8 ed 2b 82 b2 30 0b 74 ..........+..0.t
0020 9a c7 80 6c 80 18 04 fb fe eb 00 00 01 01 08 0a ...l............
0030 6d 7b db 92 0c e4 c1 d2 81 7e 00 bf 4d 45 53 53 m{.......~..MESS
0040 41 47 45 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 AGE.content-leng
0050 74 68 3a 31 39 0a 65 78 70 69 72 65 73 3a 30 0a th:19.expires:0.
0060 64 65 73 74 69 6e 61 74 69 6f 6e 3a 2f 74 6f 70 destination:/top
0070 69 63 2f 74 65 73 74 74 6f 70 69 63 2e 66 6f 6f ic/testtopic.foo
0080 0a 73 75 62 73 63 72 69 70 74 69 6f 6e 3a 30 0a .subscription:0.
0090 70 72 69 6f 72 69 74 79 3a 34 0a 6d 65 73 73 61 priority:4.messa
00a0 67 65 2d 69 64 3a 49 44 3a 63 6f 6c 69 6e 2d 33 ge-id:ID:colin-3
00b0 37 39 39 37 2d 31 34 37 33 31 34 37 32 35 32 33 7997-14731472523
00c0 36 33 2d 31 36 3a 31 3a 31 3a 31 3a 31 0a 74 69 63-16:1:1:1:1.ti
00d0 6d 65 73 74 61 6d 70 3a 31 34 37 33 31 34 37 32 mestamp:14731472
00e0 36 33 37 31 32 0a 0a 7b 22 61 22 3a 31 32 33 2c 63712..{"a":123,
00f0 22 62 22 3a 22 78 79 7a 22 7d 00 "b":"xyz"}.
Note that the packets for STOMP and WebSocket show quite strong similarities. The reason for this is that the WebSocket transport used in bowerick uses STOMP as MoM protocol. However, there are also differences due to the differences between the two transports.
Consumer: MQTT (TCP) (“mqtt://127.0.0.1:55544”)
0000 02 00 00 00 45 00 00 5a ee 45 40 00 40 06 00 00 ....E..Z.E@.@...
0010 7f 00 00 01 7f 00 00 01 d8 f8 5f bd c8 97 d2 49 .........._....I
0020 2b 9d 6d 9c 80 18 04 00 fe 4e 00 00 01 01 08 0a +.m......N......
0030 63 7e 77 da 0c e4 c2 85 32 24 00 0d 74 65 73 74 c~w.....2$..test
0040 74 6f 70 69 63 2f 66 6f 6f 00 01 7b 22 61 22 3a topic/foo..{"a":
0050 31 32 33 2c 22 62 22 3a 22 78 79 7a 22 7d 123,"b":"xyz"}
Consumer: OpenWire (UDP) (“udp://127.0.0.1:55555”)
0000 02 00 00 00 45 00 01 65 ee 44 00 00 40 11 00 00 ....E..e.D..@...
0010 7f 00 00 01 7f 00 00 01 ac 91 3c 67 01 51 ff 64 ..........<g.Q.d
0020 00 00 01 45 15 00 00 00 06 00 01 7a 01 00 21 49 ...E.......z..!I
0030 44 3a 63 6f 6c 69 6e 2d 33 37 39 39 37 2d 31 34 D:colin-37997-14
0040 37 33 31 34 37 32 35 32 33 36 33 2d 31 34 3a 31 73147252363-14:1
0050 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 ................
0060 01 65 01 00 0d 74 65 73 74 74 6f 70 69 63 2e 66 .e...testtopic.f
0070 6f 6f 01 18 00 00 00 05 00 01 7b 01 00 21 49 44 oo........{..!ID
0080 3a 63 6f 6c 69 6e 2d 33 37 39 39 37 2d 31 34 37 :colin-37997-147
0090 33 31 34 37 32 35 32 33 36 33 2d 31 36 3a 31 00 3147252363-16:1.
00a0 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 01 ................
00b0 65 01 00 0d 74 65 73 74 74 6f 70 69 63 2e 66 6f e...testtopic.fo
00c0 6f 00 00 01 6e 00 01 7b 01 00 21 49 44 3a 63 6f o...n..{..!ID:co
00d0 6c 69 6e 2d 33 37 39 39 37 2d 31 34 37 33 31 34 lin-37997-147314
00e0 37 32 35 32 33 36 33 2d 31 36 3a 31 00 00 00 00 7252363-16:1....
00f0 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 ................
0100 00 00 00 01 00 00 00 00 00 00 00 15 00 00 00 00 ................
0110 00 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 ................
0120 01 56 fe 6b 5a e0 00 01 00 00 00 13 7b 22 61 22 .V.kZ.......{"a"
0130 3a 31 32 33 2c 22 62 22 3a 22 78 79 7a 22 7d 00 :123,"b":"xyz"}.
0140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0150 00 00 00 00 00 00 01 56 fe 6b 5a e3 00 00 01 56 .......V.kZ....V
0160 fe 6b 5a f3 00 00 00 00 00 .kZ......
TCP Streams for Consumers
In addition to the raw packet data, I also print the TCP streams that contained the payload data for the consumers. Similarly to the packet data, please note that these are only the streams that contained the actual payload and I did not check if additional streams were involved during the communication. However, you can download the pcap file from which this data was taken from the bowerick repository.
Consumer: OpenWire (TCP) (“tcp://127.0.0.1:55511”)
For the OpenWire TCP stream, I manually wrapped the data either at hopefully meaningful offsets or alternatively at column 80 in order to avoid the lines getting too long.
.....ActiveMQ............
..StackTraceEnabled....PlatformDetails..NJVM: 1.8.0_92, 25.92-b14, Oracle Corpo
ration, OS: FreeBSD, 10.3-RELEASE, amd64..CacheEnabled....Host...127.0.0.1..Tcp
NoDelayEnabled....SizePrefixDisabled....CacheSize.......ProviderName...ActiveMQ
..TightEncodingEnabled....MaxFrameSize...........MaxInactivityDuration.......u0
. MaxInactivityDurationInitalDelay.......'...ProviderVersion...5.14.0
.....ActiveMQ........s......TcpNoDelayEnabled....SizePrefixDisabled....CacheSiz
e.......ProviderName...ActiveMQ..StackTraceEnabled....PlatformDetails..NJVM: 1.
8.0_92, 25.92-b14, Oracle Corporation, OS: FreeBSD, 10.3-RELEASE, amd64..CacheE
nabled....TightEncodingEnabled....MaxFrameSize...........MaxInactivityDuration.
......u0. MaxInactivityDurationInitalDelay.......'...ProviderVersion...5.14.0..
.P..~........|. ID:colin-37997-1473147252363-0:1..tcp://localhost:55511..localh
ost
...O..........x. ID:colin-37997-1473147252363-8:1. ID:colin-37997-1473147252363
-7:1..................................~...}........z. ID:colin-37997-1473147252
363-8:1............e.7ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic..
............................^......y. ID:colin-37997-1473147252363-8:1.....p.._
}........z. ID:colin-37997-1473147252363-8:1......e.
testtopic.foo......... ID:colin-37997-1473147252363-7:1.
.....................^?...,........z. ID:colin-37997-1473147252363-8:1......e.
testtopic.foo.......{.!ID:colin-37997-1473147252363-16:1..........n............
....V.kZ.....{"a":123,"b":"xyz"}.........V.kZ....V.kZ........K...W.............
n..{.!ID:colin-37997-1473147252363-16:1........n..........
Consumer: STOMP (TCP) (“stomp://127.0.0.1:55522”)
CONNECT
accept-version:1.1
host:127.0.0.1
.
CONNECTED
server:ActiveMQ/5.14.0
heart-beat:0,0
session:ID:colin-37997-1473147252363-9:1
version:1.1
.
SUBSCRIBE
receipt:2
ack:client
destination:/topic/testtopic.foo
id:1
.
RECEIPT
receipt-id:2
.
MESSAGE
content-length:19
expires:0
destination:/topic/testtopic.foo
subscription:1
priority:4
message-id:ID\ccolin-37997-1473147252363-16\c1\c1\c1\c1
timestamp:1473147263712
{"a":123,"b":"xyz"}.
ACK
receipt:3
subscription:1
message-id:ID\ccolin-37997-1473147252363-16\c1\c1\c1\c1
.
RECEIPT
receipt-id:3
.
Consumer: WebSocket (TCP) (“ws://127.0.0.1:55533”)
GET / HTTP/1.1
Host: 127.0.0.1:55533
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: QzJg5xeq9ERBZCTZLR7Ugw==
Sec-WebSocket-Version: 13
Pragma: no-cache
Cache-Control: no-cache
HTTP/1.1 101 Switching Protocols
Date: Tue, 06 Sep 2016 07:34:15 GMT
Connection: Upgrade
Sec-WebSocket-Accept: xjAqlX7Q/bGXOz0vro4ef+UXle4=
Sec-WebSocket-Protocol: stomp
Upgrade: WebSocket
...3...|...p...V.......G.......P.......@...........9...gCONNECTED
server:ActiveMQ/5.14.0
heart-beat:0,0
session:ID:colin-37997-1473147252363-9:2
version:1.2
.....n.}.,Um.'Dk.
c]..hO..i@.ArA..e...uZ..vG.@`A.doJ.^.$..~..MESSAGE
content-length:19
expires:0
destination:/topic/testtopic.foo
subscription:0
priority:4
message-id:ID:colin-37997-1473147252363-16:1:1:1:1
timestamp:1473147263712
{"a":123,"b":"xyz"}.
Consumer: MQTT (TCP) (“mqtt://127.0.0.1:55544”)
....MQTT...<..paho216318442465645 ........
testtopic/foo......2$.
testtopic/foo..{"a":123,"b":"xyz"}@...
Concluding Remarks
With respect to MoM protocols/transports, the ActiveMQ broker is capable of forwarding data to various MoM protocols/transports. bowerick adds further convenience functionality on top of this.
A simple toy example was used to illustrate the use of multiple MoM protocols/transports with bowerick. In addition, raw network packet data as well as TCP stream data of some parts of the corresponding communication were shown.
With respect to the serialization, I used my current favorite serialization approach from an inter-operation perspective: byte arrays of UTF-8 encoded JSON strings. This way of serializing data provides, in my opinion, excellent inter-operation capabilities with respect to, e.g., MoM protocols/transports as well as programming language combinations.
With respect to the MoM protocol/transport, I quite like the simplicity of STOMP. The captured STOMP TCP stream nicely shows the clarity of this protocol. On the other hand, I also liked MQTT because of its minimalism.
I hope that this post is useful. As usual, constructive criticism, comments, and suggestions are highly appreciated.
ruedigergad 
Pingback: Polyglot Messaging & Integration for Fun | ruedigergad