cli4clj Version 1.0.0 Released

In this post, I announce the release of version 1.0.0 of cli4clj. Even though I was already fairly happy about the state of cli4clj as posted before, I decided to add some more improvements. In the following, I first provide an overview over the most important features of cli4clj. Subsequently, I outline the improvements since the last release.

The most important features of cli4clj are:

  • Simple configuration via maps
  • Command line history
  • Command line editing
  • Tab-completion
    • For command names
    • Hints for selected commands: based on function arguments and custom hints
  • Aliases can be used to define alternative command names, e.g., for shortcuts.
  • Clojure data types, e.g., vector, list, map, etc., can be used as command arguments.
  • Build-in help
  • Information about function arguments is displayed as help and completion hints.
  • Customizable, similar to the Clojure REPL
  • Functionality for testing CLIs via unit tests

Since the last release, the most important additions are tab-completion and the functionality to extract and display information about function arguments, both in help and tab-completion hints. Following the aim of cli4clj to reduce the effort for implementing a CLI, both features usually do not require additional effort to be used.

Two forms of tab-completion can be distinguished. Firstly, command names can be completed via tab-completion. When multiple completion options exist, a hint with the possible options is shown. Otherwise, the command name is completed.

Secondly, so called “completion hints” can be shown for commands. These completion hints are displayed when the tab-completion is triggered for commands that are already entered. By default, cli4clj tries to display information about function arguments, if such information could be found. In addition, a tab completion hint can be defined in the configuration.

For displaying information about function arguments, in the help and the completion hints, cli4clj extracts the identifiers of the function arguments. It can do this for globally defined functions that were defined via defn and for anonymous functions that were defined with fn when the respective functions and cli4clj configuration options are defined within the (start-cli {…}) expression. Please note that, for the fn-case, it is important that the anonymous functions are defined within the configuration options and that the configuration options are defined directly in the (start-cli {…}) expression and are not stored in a local or global var.

As usual, the new cli4clj version is already available via clojars. In addition, API docs and detailed test results are available via the cli4clj github repository.

Posted in cli4clj, Libs. | Tagged , , , | 4 Comments

Tesis Doctoral: Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems

This is just a brief post to announce the publication of my tesis doctoral (doctoral thesis/PhD thesis/dissertation) entitled “Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems“. The thesis is available via as open access under the terms of the Creative Commons Attribution-ShareAlike 4.0 International License Creative Commons License.

Besides papers that were published on scientific conferences and workshops, I also published many of the evaluation prototypes as Open Source Software. For an overview of all my published papers, not only those that were published in scope of my thesis, please see the list of my publications. I also already introduced some of the respective Open Source Software projects in posts on this blog and in a brief overview of my Open Source projects and contributions. I hope to publish some more details about these prototypes here in the future as my time permits it.

For now, I conclude by citing the acknowledgments of my tesis doctoral. I am deeply thankful for all the great support I had been given:

“First of all, I thank my beloved family, especially my wonderful wife, son, parents, and siblings. While the research and work during the thesis occupied much of my time, you always reminded and helped me to also esteem the other wonderful things in life. Your love, encouragement, and support are indispensable and I am deeply grateful. I love you.

I also thank my friends for their support and for always lending a sympathetic ear. I very
much enjoyed our discussions on- and off-topic with respect to my thesis scope and I am deeply thankful for your support.

Furthermore, I thank my supervisors Inma and Martin. Learning to research and to publish research results under your supervision was an exciting and fascinating experience that I enjoyed very much. I very much value your feedback and support which were very helpful and I am greatly thankful for having you as my supervisors.

I also thank my former and current colleagues with whom I had and still have the pleasure to work with. It was and still is a great experience to work in such a creative and inspiring environment. I very much value the fruitful discussions and exchange we had and look forward to more to come.

Moreover, I thank all the members of the UCASE research group at the University of Cádiz, especially Juan, Paco, Antonio, and Lorena. You supported me with very valuable feedback and have always been fabulous hosts when I visited you in Cádiz.

I also thank all members of the PhD students round table at the Frankfurt University of Applied Sciences and my colleagues from other universities with whom I worked with in various research projects. We always had great discussions and it was very valuable to me to meet and exchange with you.

Last but not least, I thank Rodion Iafarov and Ashutosh Shirole whom I supervised as student assistants and in scope of their Masters theses. I very much enjoyed to work with you and thank you very much for your contributions.”

Posted in Announcements, Misc. | Tagged , , , , | Leave a comment

Update on cli4clj for easing the implementation of CLIs in Clojure

In my last post, I introduced my first shot on cli4clj, which is intended to ease the implementation of command line interfaces (CLIs) in Clojure. Since then, I thought a lot about the future directions of cli4clj and did various experiments for further improving it. In this post, I first give an overview over the most important improvements and afterwards outline some more details about the new implementation and on how to use cli4clj.

The major improvements since the last version are:

  • Command history
  • Command line editing
  • Tab completion
  • Functionality for testing CLIs

Command history, command line editing, and tab completion are implemented with the help of jline2. While currently the command history is not persisted across CLI runs, jline2 also allows to store the history of commands such that perspectively this feature could be added as well. Tab completion currently only considers the available commands. These features are present out of the box and nothing special needs to be implemented for activating them.

With respect to testing CLIs, I added functionality for programmatically entering input and for evaluating the corresponding output. In a sense, programmatically entering input could be also considered as “scripting” actions in the CLI. For evaluating the output, the resulting string output, either to stdout or stderr, is returned which can then be used to verify that the CLI actions resulted in the desired behavior.

Simple examples for using the testing functionality are provided in the projects test implementation. For convenience, I also include two examples below:

(deftest add-cmd-cli-interaction-stdout-test
 (let [cli-opts {:cmds {:add {:fn #(+ %1 %2)}}}
       test-cmd-input ["add 1 2"]
       out-string (test-cli-stdout #(start-cli cli-opts) test-cmd-input)]
   (is (= "3" out-string))))

(deftest add-cmd-cli-interaction-stdout-multiline-test
 (let [cli-opts {:cmds {:add {:fn #(+ %1 %2)}}}
       test-cmd-input ["add 1 2" "add 3 4" "add 5 6"]
       out-string (test-cli-stdout #(start-cli cli-opts) test-cmd-input)]
   (is (= (expected-string ["3" "7" "11"]) out-string))))

The first example only covers a single command. The second example shows how multiple commands can be handled. Input commands are defined as a vector of strings in which each string corresponds to one line of input to the CLI. The “expected-string” function is a convenience function to format the expected strings into the format as will be emitted by the test-cli-xxxxx function (The most important point here is that the platform specific line delimiter will be used.). Furthermore, depending on the test case it may be desired to evaluate the output that is printed to either stdout or stderr. The above example show the functions for stdout. Below is an example for stderr:

(deftest add-cmd-cli-interaction-cmd-error-test
 (let [cli-opts {:cmds {:div {:fn #(/ %1 %2)}}}
       test-cmd-input ["div 1 0"]
       out-string (test-cli-stderr #(start-cli cli-opts) test-cmd-input)]
   (is (= "Divide by zero" out-string))))

My main use case for CLIs is to implement them in the -main method for launching them via simple “java -jar xxxxx” calls. Thus, I implemented the testing functionality such that it can also be easily used for testing -main methods without the need to pass additional parameters to these methods etc. Examples of how this is done can be seen in which demonstrates the testing functionality for the example -main method that is included cli4clj.

Overall, I am much happier with the current state of cli4clj right now. I think that it now suits my needs much better than before. I especially like the interactive features provided by jline2 and in addition the new testing capabilities which should ease the implementation and improve the understandability of tests.

As usual, the latest version of cli4clj is available on clojars. I hope that you consider cli4clj useful. Comments and inspirations are always highly appreciated.

Posted in cli4clj, Libs. | Tagged , , , | 1 Comment

Easing the Implementation of Simple Interactive CLIs for #Clojure Applications with cli4clj

tl;dr: cli4clj is a library for easing the implementation of interactive command line interfaces (CLIs) for Clojure applications. Currently, the main aim is to allow an easy implementation of a custom interactive CLI that is easily usable, even by people unfamiliar with Clojure. cli4clj is already available via Clojars.

This post is intended as a brief announcement of cli4clj. As my time is currently, unfortunately, pretty limited I cannot elaborate in depth on all the details. So, I will try to briefly outline my motivation for creating cli4clj, its purpose, (current) features, and how to use it.

Personally, I think that interactive CLIs are great, especially for experimenting and playing around with ideas. This is maybe one reason why I like to work with Clojure and its REPL.

In the past, I already added simple interactive CLIs to some of my applications and prototypes as they allow nice and easy interaction at run-time. An example is my Clojure clj-net-pcap prototype. However, my current implementation of this particular CLI is, well, lets say, very simple and naive. While this implementation approach of the CLI was sufficient back when I started clj-net-pcap, the CLI became increasingly complex over time.

For languages like Java there are libraries like cliche or the Spring Shell that aim on easing the implementation of “in-application” interactive CLIs. Unfortunately, I could not find an equivalent for Clojure that suited my needs.

While the Clojure REPL can also be started from within any third-party application, having a full-featured REPL as in-application CLI was too much for my taste. There is some quote that, “with great power comes great responsibility.” In a sense I invert this principle for cli4clj in the way that I want cli4clj to be easily usable, from implementation and CLI-usability perspective, (without having the user to worry about great or complex duties/responsibilities) and thus purposely reduce the set of available features as compared to the default Clojure REPL (limit the powers of the CLI). The result is a simple and easily implementable interactive CLI that can also be used by people who are not familiar with Clojure.

In the following listing, a simple example is shown. In this example, a command named “test” is defined for the CLI that simply prints a test message. You can also see definitions for short and long information texts that will be printed via the integrated “help” command.

(defn -main [& args]
  (let [cli-opts {:cmds {:test {:fn #(println "This is a test.")
                                :short-info "Test Command"
                                :long-info "Print a test message."}}}]
    (start-cli cli-opts)))

In the CLI, this looks as follows:

cli# test
This is a test.

Commands can also have arguments as shown in the next example for the “add” command. In addition, this example shows the definition of an alias, “a”, for being used as a shortcut to “add”.

(defn -main [& args]
  (let [cli-opts {:cmds {:add {:fn (fn [a b] (+ a b))
                               :short-info "Add two values."}
                         :a :add}}]
    (start-cli cli-opts)))

In the CLI, this looks as follows:

cli# add 1 2
cli# a 2 3

The functions that implement the actual functionality of the commands can be globally, locally, or anonymously defined. In my opinion, the capability to use locally and anonymously defined functions is great because it, e.g., allows to easily leverage closures for the CLI functionality.

A more complex and commented example of how cli4clj is intended to be used can be found in the example in the cli4clj GitHub repository. Furthermore, you can see some example output of a CLI session using this example code in the README on GitHub.

Current features of cli4clj are, e.g.:

  • Simple configuration via a map
  • Easily usable interactive CLI
  • Allows to use globally, locally, and anonymously defined functions
  • Clojure data types for command arguments, e.g., int, float, vector, list, set, …
  • Aliases for commands enable, e.g., shortcuts
  • Help command for displaying information about available commands.
  • Based on the Clojure REPL
  • Customizable similarly to the Clojure REPL

So far, I just put a few hours into implementing and documenting cli4clj. However, I am already very happy with the result, its features, and the usability. I also already uploaded cli4clj to Clojars. In the future, I plan to further work on cli4clj and improve it. One idea could be, e.g., to support named arguments like many Linux commands do, in the way “–foo” or “-f”, and to supply default values for options that are not explicitly set.

I hope that you find cli4clj helpful. All feedback, ideas, and contributions are highly appreciated.

Posted in Announcements, cli4clj, Libs. | Tagged , , , , | 1 Comment

One Year with the Friendly Interactive Shell (fish) aka fish-shell

A little bit more than one year ago, I learned about the friendly interactive shell (fish) aka fish-shell and decided to give it a try. Before, I primarily used the Bourne Again Shell (bash) as my main shell. In this post, I write about my experience with fish and some nice tweaks I discovered.

To cut a long story short, I think that fish is really awesome and it helped me to increase my productivity a lot. I am not sure if I will remember to address all the cool things I like about it in this post but I’ll just start and see where I end up. In the following, I will just start with the points that come to mind first and I will outline the features that I consider cool about them.

Automatic Suggestions based on the Command History and Search in the History

When typing something, fish directly starts to provide suggestions of potential completions based on matching the current input to the command history. If a suggestion is useful, just press the right-arrow key and enter to execute it. Otherwise, if you encounter something that looks similar to something that you used before, you can search within the history for matching entries via the up- and down-arrow keys. This matching in the history, like most (or even all) matches in fish, not only matches the beginning of the entries in the history but matches arbitrary sub-strings.


Fish features pretty clever completions for many “standard” commands. This starts by providing hints and explanations when completing a partially entered commend and goes further to meaningfully completing the corresponding command line arguments.

Another nice completion feature is that the completion is not limited to matching from the start of a string, like bash, but instead it is capable to match sub-strings. I’ll use an example for illustrating this: Say the directory “foo/” contains the files “bar.txt” “” “bar.pdf” then one can type:
“ls foo/pdf<TAB>”, which will complete to “foo/bar.pdf”.
This is not restricted to the string at the end of the file but generally matches sub-strings.
So for “bar_abc_123.txt” “bar_def_123.txt” “bar_ghi_123.txt”
“ls foo/def<TAB>” will complete to “foo/bar_def_123.txt”.

Furthermore, you can also use completion with wildcards as well: For a directory with files “123abc.txt” “456abc.xml” “789def.pdf” “123def.pdf”.
Entering: “*a<TAB>” will complete to “*abc.” and will offer the options to complete with “txt” or “xml”.
Entering: “*d<TAB>” will complete to “*def.pdf”.

History Token Search

Another cool feature is that fish lets you go through the previously entered history token-wise. I typically use this when I need to deal with long paths or complex command line arguments. I bound the functionality history-token-search-backward and history-token-search-forward to ALT-up and ALT-down respectively as I use it quite often.

Word-wise Movement and Deletion

Yet another cool feature is the ability to perform word-wise movement and deletions when entering commands. This is particularly useful when dealing with long and complex command lines. I put backward-word, forward-word, and backward-kill-word on ALT-left, ALT-right, and ALT-backspace.

History of Visited Directories

Another neat feature is that fish maintains a history of directories that you have been to and allows you to cycle through these directories. This is done via the backward-word and forward-word functionality. As long as nothing was typed yet, using backward-word and forward-word cycles through the list of visited directories.

Git Status in the Prompt

In the Arch Linux Wiki there is also a simple guide on how to enable a nicely looking prompt that shows you the current status of a git repository whenever the current path happens to be inside of a git repository:

Colors (“Eye Candy”)

Well, you may consider this “just” “eye candy” but from my experience having nicely colored output also helped me when working with the shell. Like many of the things I discussed above, this is most likely something that you have to try on your own for some time in order to get a real idea of its benefits


For certain tasks, like writing short snippets of scripts, such as for loops etc. in the shell, be warned that fish does not follow the syntax of shells like bash or sh. It also uses a different way for handling and setting environment variables. This may look like a problem at first, however, for me all the great advantages of fish outweigh the slight overhead of getting used to this minor peculiarity by far.


I think fish is a truly great shell and that it helped me to become much much more productive. It took me some time to integrate the new capabilities into my workflow but once I got more and more used to the new features, I am really happy about having switched to fish.

In this post, I briefly describe some of the features that I think of as most valuable. All the things I posted are based on my current workflow and experience.

Please let me know, if you happen to know other cool features that I did not mention yet. I am always looking for further improving my workflow.

My Key-bindings

Last but not least, here are my, admittedly pretty simple, fish key-bindings:

~ cat .config/fish/functions/ 
bind \e, 'history-token-search-backward'
bind \e. 'history-token-search-forward'
bind \eup 'history-token-search-backward'
bind \edown 'history-token-search-forward'
bind \ebackspace 'backward-kill-word'
bind \eleft 'backward-word'
bind \eright 'forward-word'
Posted in Snippets | Tagged , , | Leave a comment

Dynamic Distributed Remote Packet Capturing with DRePCap

In scope of my PhD as well as my work as researcher, among other things, I also researched the concept of dynamic distribute remote packet capturing. Some results of this work were published in the paper: “Monitoring Traffic in Computer Networks with Dynamic Distributed Remote Packet Capturing,” Ruediger Gad, Martin Kappes, Inmaculada Medina-Bulo, 2015 IEEE International Conference on Communications (ICC), London, UK.

For the evaluation of this approach, I developed the Distributed Remote Packet Capturing (DRePCap) prototype, which was released as Open Source Software. The empirical results in the paper mentioned above were obtained with this prototype.

Other approaches for remotely capturing packets are, e.g., rpcap or packet capturing via SSH. However, these approaches are typically comparably simple in the way that they only explicitly consider single sensor operation and point-to-point connections between remote packet capturing sensor and data receiver. The concept of dynamic distributed remote packet capturing aims on providing more advanced functionality like:

  • providing an overarching infrastructure for network wide remote packet capturing that can be used by an arbitrary number of receivers and with an arbitrary number of sensors,
  • dynamic configuration of packet capturing sensors,
  • dynamic routing of the captured packet data to potentially multiple destinations,
  • or easing packet capturing at multiple remote sensors at the same time.

Furthermore, for improving the performance and usability, the concept is also capable to:

  • employ cooperative sensors for improving the packet capturing performance,
  • self-adaptive sampling for avoiding overload situations at individual sensors,
  • and self-adaptive sensor cooperation in order to ease the usability of operating multiple sensors.

DRePCap consists of four general components:

In the following figure, an overview of the DRePCap architecture is shown:


In this post, I just briefly present the high-level overview of the dynamic distributed remote packet capturing concept and of DRePCap. For more details, please refer to the paper mentioned above, for now. As time permits, I will try to add some more posts about DRePCap as well as the associated sub-projects.

Posted in DRePCap, Libs. | Tagged , , , , | Leave a comment

clj-net-pcap – Improvements and Preparations for next Release

It’s been a while since I posted the last update about clj-net-pcap. Before the work on my PhD thesis started to occupy most of my time, I actually put some more effort into further optimizing and improving clj-net-pcap. However, due to my PhD thesis, I didn’t write about this progress earlier.

In this post, I briefly summarize the work and progress that had been done. In the following, an overview of the new improvements and optimizations is given:

  • Optimized raw data acquisition performance (by a factor of about 5.9)
  • Alternative method for extracting information from raw packet data (performance improvement by a factor of about 2.5)
  • Domain Specific Language (DSL) for dynamically configuring the data extraction
  • Prototype of a self-adaptive mechanism for adjusting the DSL-based data extraction depending on performance constraints
  • Functionality to write extracted data to files and named-pipes (fifos)
  • Multiple output formats (CSV, JSON, ARFF)

The first four of these improvements were also published in the paper: “Improving Network Traffic Acquisition and Processing with the Java Virtual Machine,” Ruediger Gad, Martin Kappes, and Inmaculada Medina-Bulo, 20th IEEE Symposium on Computers and Communications (ISCC), in press. The paper also contains an evaluation of these improvements. The exact versions of clj-net-pcap that were used for the work presented in this paper are marked with corresponding tags.

The performance optimizations of the raw data acquisition performance were done by improving the forwarding of raw packet data from the native libpcap-based parts to the parts inside the JVM. These optimizations include patches in the corresponding JNI-related C and Java code of jNetPcap that is used for interconnecting the native and JVM-based parts. These patches are released under the same license as jNetPcap and are available in the clj-net-pcap repository.

Before the new release will be published, there is still some housekeeping that has to be done. However, the new functionalities can already be used now.

Posted in clj-net-pcap, Libs. | Tagged , , , , | Leave a comment