In my previous post, I already outlined the problem of testing multi-threaded interactive command line interfaces (CLIs). In the meantime, I came up with some improvements for addressing the testability of multi-threaded command line applications with cli4clj.

In this post I will cover:

• a recap of the problem of testing interactive CLIs vs. multi-threading,
• a first improved solution by synchronizing the testing via CLI output with string-latch,
• and an enhanced string-latch with callback functions.

Testing Interactive CLIs vs. Multi-threading

As long as the functionality associated with a command in a CLI is executed in the main CLI UI thread, testing a CLI is straight forward. One can simply define the commands that are to be executed during a test and, as long all functionality is executed in the main CLI UI thread, the CLI output will be in the order in which the commands are defined. Furthermore, all commands will be completely executed before the CLI UI main thread and with this the test function call terminates. An example of such a test is shown in the following listing:

(test/deftest advanced-main-method-example-test
  (let [test-cmd-input ["add 1 2"
        "divide 4 2"
        "d 3 2"]
        out-string (cli-tests/test-cli-stdout
                     #(example/-main "")
                     test-cmd-input)]
    (test/is (=
               (cli-tests/expected-string ["3" "2" "3/2"])
               out-string))))

However, when the functionality associated with a command is executed in another thread, testing becomes more complicated. In the following listing a contrived example is shown:

(defn- async-test-fn
  []
  (println "Starting...")
  (let [tmp-out *out*]
    (doto
      (Thread.
        (fn []
          (binding [*out* tmp-out]
            (utils/sleep 500)
            (println "Finished."))))
      (.start)))
  (println "Started."))

(test/deftest async-cmd-not-finished-test
  (let [cli-opts {:cmds {:async-foo {:fn async-test-fn}}}
        test-cmd-input ["async-foo"]
        out-string (cli-tests/test-cli-stdout
                     #(cli/start-cli cli-opts)
                     test-cmd-input)]
    (test/is (=
               (cli-tests/expected-string
                 ["Starting..." "Started." "Finished."])
               out-string))))

The async-cmd-not-finished-test will typically fail. The reason for this is that the execution of async-test-fn spawns a thread in which the “actual processing” (mimicked via utils/sleep) is done. The main CLI UI thread, however, is not aware of this thread and thus only blocks until the test function has terminated.

Consequently, cli-tests/test-cli-stdout will typically terminate before the processing thread has finished. Thus, the output of “Finished.” will typically be not included in out-string such that the test will fail. Even worse, this can result in a race condition with seemingly random behaviour in which tests pass sometimes and sometimes not.

The stop-gap solution introduced back then was to use a hidden _sleep command to cause the main CLI UI thread to wait until the processing thread had finished. However, using sleeps for this kind of use case is not a good solution.

Synchronized Testing via CLI Output with string-latch

In order to improve the testability of multi-threaded CLIs, the execution of non-UI threads has to be synchronized with the test-execution of the main CLI UI thread. As this is about CLIs, one natural option for this synchronization is to use the CLI output for synchronization. In the end, even the execution of multi-threaded functionality in an interactive CLI can be typically expected to cause some sort of CLI output.

For synchronizing the test execution based on the CLI output, I added what I call “string-latch”. The name is inspired by the Java CountDownLatch, which is used under the hood. In its simplest form, a string-latch is defined as a vector of strings that are expected to occur in the given order. Thereby, it is sufficient to only define key elements of the expected output such that it is not required to exactly reproduce the actual test CLI output. In the following listing, the string-latch solution for the above example problem async-cmd-not-finished-test is given:

(test/deftest async-cmd-string-latch-stdout-test
  (let [cli-opts {:cmds {:async-foo {:fn async-test-fn}}}
        test-cmd-input ["async-foo"]
        sl (cli-tests/string-latch ["Finished." "\n"])
        out-string (cli-tests/test-cli-stdout
                     #(cli/start-cli cli-opts)
                     test-cmd-input
                     sl)]
    (test/is (=
               ["Starting..." "\n" "Started." "\n" "Finished." "\n"]
               (sl)))
    (test/is (=
               (cli-tests/expected-string
                 ["Starting..." "Started." "Finished."])
               out-string))))

In comparison to the problematic test async-cmd-not-finished-test, the changes are: the addition of the string-latch “sl” and the use of the test-cli-stdout that takes the string-latch as argument.

String Latch with Callback Functions

As another improvement, I added the possibility to execute callback functions for string-latch entries. These callback functions will be executed when the defined string-latch entry was matched. In order to define string-latch entries with callbacks, a vector containing the match-string and the callback is used as string-latch entry instead of a simple string. The following listing shows an example that builds up on the previous examples:

(test/deftest async-cmd-string-latch-stdout-with-callback-test
  (let [cli-opts {:cmds {:async-foo {:fn async-test-fn}}}
        test-cmd-input ["async-foo"]
        val-0 (atom nil)
        val-1 (atom nil)
        val-2 (atom nil)
        sl (cli-tests/string-latch
             [["Starting..." #(reset! val-0 %)]
              ["Started." (fn [v] (reset! val-1 v))]
              ["Finished." #(reset! val-2 %)]
              "\n"])
        out-string (cli-tests/test-cli-stdout
                     #(cli/start-cli cli-opts)
                     test-cmd-input sl)]
    (test/is (= ["Starting..."] @val-0))
    (test/is (= ["Starting..." "\n" "Started."] @val-1))
    (test/is (=
               ["Starting..." "\n" "Started." "\n" "Finished."]
               @val-2))
    (test/is (=
               (cli-tests/expected-string
                 ["Starting..." "Started." "Finished."])
               out-string))))

The string and the extended, vector with callback, versions of string-latch entries can also be mixed. This is shown in the following listing:

(test/deftest async-cmd-string-latch-stdout-with-mixed-callback-test
  (let [cli-opts {:cmds {:async-foo {:fn async-test-fn}}}
        test-cmd-input ["async-foo"]
        val-0 (atom nil)
        val-1 (atom nil)
        val-2 (atom nil)
        sl (cli-tests/string-latch ["Starting..."
                                    ["Started."]
                                    ["Finished." #(reset! val-2 %)]
                                    "\n"])
        out-string (cli-tests/test-cli-stdout
                     #(cli/start-cli cli-opts)
                     test-cmd-input
                     sl)]
    (test/is (= nil @val-0))
    (test/is (= nil @val-1))
    (test/is (=
               ["Starting..." "\n" "Started." "\n" "Finished."]
               @val-2))
    (test/is (=
               (cli-tests/expected-string
                 ["Starting..." "Started." "Finished."])
               out-string))))

Summary

In this post, I briefly introduced the string-latch for testing interactive CLIs and its most important features. I think the string-latch is an important improvement in comparison to the previous _sleep stop-gap solution. For my use cases, the string-latch works very well so far. I hope to get the time to write a bit more about the details of the string-latch in future posts.

As usual, you can get the latest version of cli4clj from clojars.org and I would be happy if you left some constructive feedback or comments.