Performance of “Method Calls” on “Clojure Objects”

tl;dr

There are various ways for implementing objects or object-like constructs with Clojure. Besides differences with respect to implementation aspects, these ways also differ with respect to their performance. In this post, I write about the results of some simple benchmarks for evaluating the performance of some approaches that I could find in books and the Internet. If you are just interested in the overview, just scroll down to the performance overview plot that is located close to the end of this post.

Introduction and Motivation

For modelling data, I actually like generic data types such as maps, vectors, etc. and Clojures functionality for dealing with these generic data types very much. However, in certain circumstances, I like to use “classical objects” in which I encapsulate certain elements and that provide methods for interacting with the encapsulated entities.

An example of a situation in which I like to use “classical objects” is when a resource, e.g. a connection, has to be created and various operations need to be performed with this resource. Examples of such resources are: network sockets, JMS connections, or database connections. Examples of operations that I want to perform with these resources are: transmitting data, acquiring performance statistics, or closing the resource.

While there are already various posts etc. about the implementation or engineering aspects related to using object-like functionality in Clojure, I missed a comparison of the corresponding performance aspects. I think that, to certain extend, the choice which approach is used or if object-like behavior is used at all is a matter of taste. However, taking the aforementioned examples, such as communication or database connections, performance is an important aspect.

In the remainder of this post, I first outline the ways for implementing object-like behavior that were analyzed. Afterwards, the results of the performance benchmarks are shown.

Ways for Implementing Object(-like) Functionality in Clojure

The intention of this section is not to provide an in-depth discussion of ways for implementing object-like functionality in Clojure. The aim is rather to provide an overview of the working principles of the analyzed methods.

defprotocol and deftype/defrecord

The use of defprotocol in conjunction with deftype/defrecord is, e.g., discussed in:
https://dzone.com/articles/object-oriented-clojure
http://thinkrelevance.com/blog/2013/11/07/when-should-you-use-clojures-object-oriented-features

In the following listing, extracts for an example of this approach using defprotocol and defrecord are shown. The working principles for deftype are similar.

(defprotocol WithConnection
 (send [this data])
 (close [this]
 ...))

(defrecord ConnectionWrapper [resource]
 WithConnection
 (send [_ data] (.send resource data))
 (close [_] (.close resource))
 IFn
 (invoke [this data] (send this data)))

(defn create-connection
 [...]
 (let [resource (...)]
 (->ConnectionWrapper resource)))

(...
 (let [c (create-connection ...)]
 ; Send via "send" method defined in WithConnection.
 (send c "my data")
 ; By implementing IFn the object itself can also be used as a function.
 (c "my data")
 (close c)))

In the example, a “resource” is wrapped in an object and the object provides methods for interacting with the resource. In addition to the definition of the “send” method, the example also shows that a record can implement IFn such that the corresponding objects can be used as functions.

Closure with Functions in a Map

In “The Joy of Clojure” by Michael Fogus and Chris Houser, Manning, 2011, pp. 139-140, an example of implementing object-like functionality via closures and a returned map with functions is given. In “Let over Lambda” by Doug Hoyte, 2008, pp. 32-33, a similar way for implementing object-like functionality by using a closure and returning a list of multiple functions is given.

In the following listing, extracts for an example of this approach are shown:

(defn create-connection
 [...]
 (let [resource (...)]
 {:send (fn [arg] (.send resource arg))
 :close (fn [] (.close resource))
 ...}))

(defn send
 [conn arg]
 ((conn :send) arg))

(defn close
 [conn]
 ((conn :close)))

(...
 (let [c (create-connection ...)]
 (send c "my data")
 (close c)))

The example follows the same usage scenario as the previously given scenario for defrecord. In order to ease the method calls, additional functions are used to hide the map-based implications. Instead of functions, macros could be used as well but I just show functions here for illustration purposes.

Closure with Function Dispatch based on Keywords

In http://pramode.net/clojure/2010/05/26/creating-objects-in-clojure/ and “Clojure in Action” by Amit Rathore, Manning, 2012, pp. 326-328, closures and function dispatch based on keywords are used for mimicking object-like behavior. While I have to confess that I also used similar implementations in some of my older projects, I would not use this approach anymore. Seen from today’s perspective, I consider this as a rather complicated and inflexible way for achieving object-like functionality. Furthermore, the results of the performance evaluation shown in this post also indicate that, depending on the actual setting, the performance is inferior compared to the other approaches.

In the following, an example for this way of achieving object-like functionality is shown:

(defn create-connection
 [...]
 (let [resource (...)]
 (fn [arg]
 (condp =
 :close (.close resource)
 ...
 (.send resource arg)))))

(defn close
 [conn]
 (conn :close))

(...
 (let [c (create-connection ...)]
 (c "my data")
 (close c))

The example follows the same usage scenario as the other two examples.

Performance Result

In the following, at first, the tested scenarios are outlined. Subsequently, plots showing the results of the benchmarks are shown and described briefly. Afterwards, the procedure of how the benchmarks were performed is summarized and links to the benchmark code and raw data are given. The entire benchmark code including the post-processing scripts is available as Open Source Software, so you can give it a shot on your own.

Tested Scenarios

Below, the tested scenarios are outlined.
For more details, please have a look at the source code for which links are given below.

The tested scenarios can be roughly distinguished as follows:

  • baseline measurements for providing a basis for comparisons,
  • measurements for the approach based on keywords and condp (condp-x),
  • measurements for the map-based approach (map-x),
  • and measurements for the deftype and defrecord based approaches (record-x and type-x).

The baseline measurements tested the following scenarios:

  • adding two constants (baseline-0-add),
  • calling an fn returning a constant (baseline-1-fn-const),
  • calling an fn with an argument returning the argument as-is (baseline-2-fn-arg),
  • calling an fn that adds a constant value to the fn argument (baseline-3-fn-add-arg),
  • and calling an fn that uses a closure to add a pre-defined value to the fn argument (baseline-4-fn-add-closure).

For the condp-based scenario, the number of tests of the condp expression were varied. The actual operation that was to be performed was in the final default clause of the condp expression. The value x in the scenario description condp-x matches the number of tests that preceded the final default clause.

For the map-based tests, the number of key-value pairs in the map was varied. The value x in map-x is the number of key-value pairs that were added in addition to the actual entry that was to be executed. For map-x, the map was used as fn and the keyword as argument: (m :kw). For map-rev-x, the keyword was used as fn and the map as argument: (:kw m).

For the defrecord and deftype based approaches, the following situations were considered:

  • calling the fn defined in the implemented protocol as function and passing the object etc. as arguments: (f obj arg)
  • calling the fn defined in the implemented protocol as method of the object: (.f obj arg)
  • calling the object as fn (The type/record implements IFn.) and using the fn-way for forwarding the call to the actual processing functionality.
  • calling the object as fn (The type/record implements IFn.) and using the method-way for forwarding the call to the actual processing functionality.
  • calling an fn that wasadded by extending the protocol.

Result Plots

In the figure below, an overview of the absolute execution time is shown. Please not that the y-axis uses a logarithmic scale. The shown values are the mean values as reported by Criterium. In addition, the standard deviation is shown with error bars. However, due to the small standard deviation, these error bars are hardly visible.

2016-04-12_20-55_colin_FreeBSD_out_1

In the figure below, an overview of the relative execution time of the same result data as shown in the previous figure is shown. The result of scenario “baseline-4-fn-add-closure” was used as reference for the calculation of the relative values. Please note that the y-axis is linear and that the y-axis range was purposely chosen to focus on the majority of results at the expense of accepting that some values could not be fit on the plot.

2016-04-12_20-55_colin_FreeBSD_out_2

Benchmark Procedure and Source Code

For the execution of the benchmarks, criterium (https://github.com/hugoduncan/criterium) was used. Criterium automatically takes care of warm-up, repeated measurement execution, statistics calculation etc. Please note: even though the source code contains “quick-bench” right now, the benchmarks were done with “bench” (In the meantime, I changed the default to “bench”.).

Each benchmark scenario was implemented as separate Clojure test and the tests were all run via Leiningen. The source code of the benchmarks is available as Open Source Software: https://github.com/ruedigergad/clojure-method-call-benchmark/

Concluding Remarks

The evaluated scenarios were purposely chosen to be comparably simple. There may also be many more variations that could be assessed, such as other ways for implementing object-like behavior, the impact of type hints and reflection, or permutations of these aspects. However, for me, I found these results already quite interesting and I hope that the results or the benchmark code itself are useful for others as well. Comments and suggestions are always appreciated.

 

 

 

Advertisements
This entry was posted in Misc., Other Software, Snippets and tagged , . Bookmark the permalink.

2 Responses to Performance of “Method Calls” on “Clojure Objects”

  1. Alex Miller says:

    I haven’t looked at this deeply, but some comments from a quick skim…

    * In “IFn (invoke [this data] (send this data)))” you should probably call (.send resource data) instead – otherwise you are adding an additional needless invocation.

    * All of the tests using + are going to be significantly affected by boxed arithmetic – you are taking a boxed number, doing boxed +, then returning a boxed number. Typically boxed arithmetic is ~100x slower than non-boxed arithmetic. It might just be safer to pick some other operation that does not have these effects.

    * I also see in your code that you’re using quick-bench – for timings this small I really recommend using bench instead – it takes a lot longer but I find the results are significantly different sometimes.

    • ruedigergad says:

      Hi Alex,

      thanks a lot for your constructive comments!

      I implemented “IFn invoke” to just forward to the actual implementation in order to mimic a situation in which some more complex functionality is implemented in a different function and for which “IFn invoke” is just added as convenience alternative. But you are right that this adds one more (potentially superfluous) indirection. I think that depending on the situation, e.g., for simple functionality as shown in the example, duplicating the functionality from the send function in invoke or solely implementing the functionality in invoke and removing the send function could be reasonable. What do you think about this?

      For future work, I will definitely keep your comment regarding the overhead of boxed types in mind. For the current example, I think that the differences in the execution times are still visible even despite of this overhead.

      Regarding quick-bench: I actually used “bench” for the benchmarks from which the shown results are taken. I just changed back to “quick-bench” to speed up the development and testing. While I wanted to point this out in the text, I seem to have forgotten to do so. Thanks for your heads up! For now, I put a brief comment about this in the text. I also consider replacing “quick-bench” with “bench” in the committed code or at least adding another comment about this to the code in order to avoid serving as a bad example.

      Thanks a lot again!

      Ruediger

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s