Bmark - A benchmarking tool for elixir - Part 2

Bmark has been released and is available on Hex.pm!

You can find the source for the release and instructions on Bmark’s github repo. As I mentioned before the blog posts trail the development. So you can expect 2 more posts after this one describing the release process.

In this post I’ll continue describing my path to release. Recall that Bmark is a tool for writing and measuring benchmarks in Elixir. In terms of this week’s tasks, recall the TODO List from last week:

~~Clean up the code~~
Configurable number of runs
Use builtin Elixir server module for Bmark.Server?
Add a readme
Add a license
Create a hex package
Test
Release

Configurable number of runs

First I need a test, I’ll write this as an end-to-end test:

#!/bin/bash

. end_to_end.sh "It accepts a count of runs in 'bmark' tests"

EXPECTED_OUTPUT=results/example.count.results

function main() {
  cd ../../
  cleanup
  run_example
  verify
}

function cleanup() {
  rm -f $EXPECTED_OUTPUT
}

function run_example() {
  mix bmark example | grep ":count test running 5 times" > /dev/null || fail "Did not run test"
}

function verify() {
  [ -f $EXPECTED_OUTPUT ] || fail "No results file $EXPECTED_OUTPUT"
  wc -l $EXPECTED_OUTPUT | grep "5" > /dev/null || fail "Wrong number of results"
  pass
}

main

This is really just a copy of bmark_runnner_test.sh with the expected count changed to 5. I should consider extracting out the common code if I have a need to make any more tests like this.

The test fails to run the benchmark:

FAIL: It accepts a count of runs in 'bmark' tests Did not run test

So I add a new bmark:

bmark :count do
  IO.puts ":count test running 5 times"
end

and run my tests again:

FAIL: It accepts a count of runs in 'bmark' tests Wrong number of results

Now I actually have to add the functionality to control the number of runs.

First I have to decide on the interface. I could simply pass an argument like this:

bmark :benchmark, 5 do
  IO.puts ":count test running 5 times"
end

This seems easy to implement but not very readable. I could use a named argument like this:

bmark :benchmark, runs: 5 do
  IO.puts ":count test running 5 times"
end

but I don’t think Elixir has them. The next best thing might be a map:

bmark :benchmark, %{runs: 5} do
  IO.puts ":count test running 5 times"
end

This is clearer but but extra map syntax is a little heavy. I could use a module attribute like ExUnit does:

@runs 5
bmark :benchmark do
  IO.puts ":count test running 5 times"
end

I think I would still prefer a named argument if it were possible. I’d love to hear which syntax you prefer. Let me know in the comments. For now I will proceed with the @runs syntax. But, how do I implement it?

I used ExUnit.Case as an example. The function __on_definition__ does the work for test in ExUnit. The key is to use Module.get_attribute to lookup the @runs attribute. Then, at the end of the bmark macro use Module.delete_attribute to delete the attribute so it can be defined again for the next bmark.

I ended up with this code:

def add_bmark(module, name) do
  with_runs(module, fn
    runs -> Bmark.Server.add(module, name, runs)
  end)
end

defp with_runs(module, f) do
  runs = case Module.get_attribute(module, :runs) do
    nil -> 10
    val -> val
  end

  f.(runs)

  Module.delete_attribute(module, :runs)
end

I chose to write with_runs as a higher order function. I handles processing @runs and deleting it at the end. In the middle it runs a user supplied function. add_bmark supplies the user supplied function which registers the bmark do block with the server.

I also made sure to add another test using another bmark without @runs to make sure it defaulted to 10 runs.

Wait, I could have used Keyword lists! They have the nice property of dropping the heavy syntax when used as the last parameters of a function. I could have:

bmark :benchmark, [runs: 5] do
  IO.puts ":count test running 5 times"
end

or even better:

bmark :benchmark, runs: 5 do
  IO.puts ":count test running 5 times"
end

This also seems like a much simpler implementation. I tried implementing this but it doesn’t seem to work. I wrote:

defmacro bmark(name, keywords) do
  runs = Keyword.get(keywords, :runs, 10)
  body = Keyword.get(keywords, :body, nil)
  quote bind_quoted: binding do
    Bmark.add_bmark(__ENV__.module, name, runs)
    def unquote(name)(), do: unquote(body)
  end
end

This allows me to write:

bmark :benchmark, runs: 5, do: IO.puts ":count test running 5 times"

but not the multiline version. I get an error:

** (CompileError) bmark/example_bmark.ex:8: undefined function bmark/3
    (stdlib) lists.erl:1352: :lists.mapfoldl/3
    (stdlib) lists.erl:1353: :lists.mapfoldl/3
    (elixir) src/elixir_exp.erl:49: :elixir_exp.expand/2
    (elixir) src/elixir.erl:206: :elixir.quoted_to_erl/3
    (elixir) src/elixir.erl:175: :elixir.erl_eval/3

So, I guess that means do ... end is always converted into a keyword list of its own. It is not merged in with a previous keyword list if it exists.

I’ll go ahead and keep the @runs syntax.

TODO List

~~Clean up the code~~
~~Configurable number of runs~~
Use builtin Elixir server module for Bmark.Server?
Add a readme
Add a license
Create a hex package
Test
Release

Use builtin Elixir server module for Bmark.Server?

When I looked over the Bmark.Server code I thought maybe I could use one of the Elixir provided server types instead of GenServer to simplify this code.

I could use Agent. But I don’t think it will simplify anything. Here’s my reasoning:

I could use Agent.update to update the list. This would mean

def add(module, name, runs) do
  GenServer.cast(:Bmark, {:add, %{module: module, name: name, runs: runs}})
end

would become

def add(module, name, runs) do
  Agent.update(:Bmark, fn (state) -> [ %{module: module, name: name, runs: runs} | state ] end)
end

Neither is easy to read. The name of the function, add, which helps make the code more understandable.

With an Agent there would be a couple of ways to implement run_benchmarks

def run_benchmarks do
  list = GenServer.get(:Bmark, fn (state) -> state end)
  {Enum.map(list, &collect_single_benchmark/1), list}
end

def run_benchmarks do
  list = GenServer.get(:Bmark, fn (state) -> { Enum.map(state, &collect_single_benchmark/1), state } end)
end

The second version is truer to the original GenServer based version but is harder to read. The first version is a bit more straightforward.

Overall, I don’t think switching to Agent would buy much so I’ll leave the code as is.

There is also the ExActor DSL which allows for simpler GenServer implementations. It is well described in Actors in Erlang/Elixir. I think this could simplify Bmark.Server. The only issue is that I am not sure I want to introduce a depdency on a third party library at this time. What do you think about this? Should I use ExActor to simplify this code or leave it as is? For now, I’ll leave it but I can always come back to revisit this code later.

~~Use builtin Elixir server module for Bmark.Server?~~

While looking at ExActor I noticied that it allows something I didn’t think I could do in Elixir. From “A more involved example:”

:runs 5 syntax

defmulticall set(key, value), state: cache_name do
  store(cache_name, key, value)
  reply(:ok)
end

The defmulticall has a keyword argument list at the end that has state: cache_name and a do block. This is similar to the bmark syntax I wanted:

bmark :name, runs: 5 do
  IO.puts "A benchmark"
end

Looking at the implementation:

defmacro defmulticall(req_def, options \\ [], body \\ []) do
  do_defmulticall(req_def, options ++ body)
end

Ah, of course, I just have to have a default value for the options argument. I can even use my default count in the default value:

defmacro bmark(name, options \\ [runs: 10], [do: body]) do
  runs = Keyword.get(options, :runs)
  quote bind_quoted: binding do
    Bmark.Server.add(__ENV__.module, name, runs)
    def unquote(name)(), do: unquote(body)
  end
end

I tried this out and it works just the way I want. After making the change I was able to remove the bmark_after_count_test.

Next steps

I’m very happy to have made the runs: 5 syntax work but also I value the effort I put into the @runs 5 syntax. I learned a lot about handling module attributes and Learning Elixir is what this blog is about.

In the coming week’s I’ll continue to work on releasing bmark. While reading over Elixir code (like ExUnit and other core macros) I realized I need more documentation in my code. I’ll add that to the to-do list and will look at adding it next week.