ClojureScript Performance Measurement
When optimizing code, measurement is critical. This post details a technique I've found useful when optimizing ClojureScript code.
Unfortunately, ClojureScript doesn't have a nice tool like Criterium. But, it does have facilities to fundamentally time how long code takes to execute.
The easiest to use is the time macro. (This also exists in Clojure, by the way.) It is really easy to use directly in the REPL:
(time (apply + (range 1e7)))
This will print out how much time was used, and return the value.
Oftentimes, you want to measure something that, in and of itself, doesn't take very much time to execute. For example, let's say you were interested in keyword lookup in maps.
(time (:a {:a 1}))
This is too quick to get a sense of what is going on. So, you could have the expression executed many times, for example, by using dotimes:
(time (dotimes [_ 10000000] (:a {:a 1})))
But, then you are also measuring the construction of the map {:a 1}. You could then improve this by leting the map value outside the dotimes, but there exists a handy tool in the standard ClojureScript library that takes care of the let and dotimes, all in one macro, simple-benchmark:
(simple-benchmark [m {:a 1}] (:a m) 10000000)
This has the benefit of printing a nice summary of what was measured:
[m {:a 1}], (:a m), 10000000 runs, 1653 msecs
But, of course, using this in the REPL presents two problems:
- You are running things in
:nonemode. Your final artifact may very well be compiled in:advancedmode where it could exhibit very different performance characteristics. - You are only measuring the performance for the particular JavaScript environment in which your REPL is running. (You may be interested in how a chunk of code behaves across a variety of environments.)
Hence we get to the main point of this post: I reach for the benchmarking setup that comes with the ClojureScript compiler repository: If you set up multiple JavaScript environments as documented at Running the Tests, and clone https://github.com/clojure/clojurescript, then within that repo you can run script/benchmark to have benchmarks compiled in :advanced mode and then run across all of your VMs.
When doing this, all of the benchmarks defined in benchmark/cljs/benchmark_runner.cljs will be executed. What I do at this point is to simply delete all of the benchmarks in that file, and add my own, written using simple-benchmark. Here is what you will see with the above benchmark defined:
Benchmarking with V8
[m {:a 1}], (:a m), 10000000 runs, 200 msecs
Benchmarking with SpiderMonkey
[m {:a 1}], (:a m), 10000000 runs, 263 msecs
Benchmarking with JavaScriptCore
[m {:a 1}], (:a m), 10000000 runs, 188 msecs
Benchmarking with Nashorn
[m {:a 1}], (:a m), 10000000 runs, 1243 msecs
Benchmarking with ChakraCore
[m {:a 1}], (:a m), 10000000 runs, 392 msecs
This runs pretty quickly, and you can then make note of the amount of time consumed on each VM, perhaps make a change to your baseline code, and re-run things to see if it runs faster, calculating speedup ratios, etc.
One thing to watch out for when running tests this way is that, if the expression you are benchmarking is sufficiently simple, Google Closure might optimize it all away. An example:
(simple-benchmark [] (clojure.string/capitalize "aBcDeF") 1000000)
If you run this, you may see that it is literally taking no time. If you take a look at builds/out-adv-bench/core-advanced-benchmark.js, you will see that the input string "aBcDeF" is no longer even in the JavaScript. I've found that you can avoid this at times by introducing a little indirection. By doing the following you will see that the JavaScript is actually applying capitalize to your string, and the benchmarks take a non-zero amount of time:
(simple-benchmark [s "aBcDeF" f clojure.string/capitalize] (f s) 1000000)
Hope this helps!