Bmark - A benchmarking tool for elixir

Last week I finished my series of posts on optimizing Elixir. During the course of those posts I developed a tool called bmark that allowed me to write benchmarks for my project, run those benchmarks, and compare results across runs. This post will be the first in a series of posts on preparing bmark as a public release through the Hex package manager.

My bmark tool can be found on github. The github repository may be ahead of these posts. My blogging workflow is to do the work and take notes. Then I edit the notes to write up the blog posts. The posts therefore trail the code a bit.

Where to get started

So I have a pile of code that I call bmark. It was good enough to do some benchmarking of my Game of Life implementation. But now I need to turn it into a useful package that anyone cane use. I’ll need to keep a list of to-do items that I need to complete for the release. Off the top of my head I need to:

TODO List

• Clean up the code
• Add a readme
• Add a license
• Create a hex package
• Test
• Release

Clean Up

First, I look over the code. It is a bit of a mess, some of it is hard to follow. I’ll start by cleaning it up a bit.

It’s been some time since I looked at the code so I’m asking myself “what was this code?”. I’ll go through the code and draw up a map:

• bmark
  • result_formatter - formats the tables of data for the results
  • server - server process to collect bmark entries and run on exit
  • distribution - t distribution for statistical T-testing of the results.
• mix/tasks
  • bmark - runs a benchmark and records results
  • bmark_cmp - compares two results

Formatter

Wait, the result_formatter doesn’t format the reuslts, it formats the output for the comparison during the bmark.cmp task. It’s name is wrong and misleading. I’ll start by cleaning it up.

I want to call the module Bmark.ComparisonFormatter. So, first I rename the test, test file and the use Bmark.ComparisonFormatter in the tests. This gives me an error telling me what to do next:

  1) test it formats lists of different sizes (Bmark.ComparisonFormatterTest)
     test/bmark_comparison_formatter_test.exs:27
     ** (UndefinedFunctionError) undefined function: Bmark.ComparisonFormatter.format/2 (module Bmark.ComparisonFormatter is not available)
     stacktrace:
       Bmark.ComparisonFormatter.format(["left", "right"], [["1", "2"], ["3", "4", 5]])
       test/bmark_comparison_formatter_test.exs:31

I need to rename the formatter. I do it but, now the end-to-end tests fail:

End-to-end Tests:
** (UndefinedFunctionError) undefined function: Bmark.ResultFormatter.format/2 (module Bmark.ResultFormatter is not available)
    Bmark.ResultFormatter.format(["test/end_to_end/test_data/similar1.results", "test/end_to_end/test_data/different.results"], [["34", "8", "6", "5", "9", "7", "6", "4", "4", "4"], ["451", "400", "422", "390", "412", "415", "399", "425", "375", "444"]])
    lib/mix/tasks/bmark_cmp.ex:45: Mix.Tasks.Bmark.Cmp.report_results/1
    lib/mix/tasks/bmark_cmp.ex:12: Mix.Tasks.Bmark.Cmp.run/1
    (mix) lib/mix/cli.ex:55: Mix.CLI.run_task/2
    (elixir) src/elixir_lexical.erl:17: :elixir_lexical.run/3
    (elixir) lib/code.ex:316: Code.require_file/2

I update the Bmark.Cmp task module and the tests pass. I have successfully renamed the formatter. But the code is still hard to follow. I do some refactoring to reorganize the code and rename some functions.

Done! Bmark.ComparisonFormatter is nice and clean now.

server

This code looks ok right now. It looks like a genserver.

However, looking at the code, I don’t like

@number_of_runs 10

This should be configurable some how. Right now I am cleaning up code so I will note this annoyance and come back to it later.

Another thought - can I use when of Elixir’s servers, like Agent, to simplify my code? This could make the code cleaner and help better communicate its intentions. Again, I add this to the to-do list for later.

TODO List

• Clean up the code
• Configurable number of runs
• Use builtin Elixir server module for Bmark.Server?
• Add a readme
• Add a license
• Create a hex package
• Test
• Release

distribution

This code looks good already.

bmark.exs

The last two functions in Mix.Tasks.Bmark could use some cleanup. I’ll start with:

defp report_single_result({module, name, list_of_times}) do
  mod = simplify_module_name(module)
  File.open("results/#{mod}.#{name}.results", [:write], fn(file) ->
    Enum.map(list_of_times, &IO.puts(file, &1))
  end)
end

This violates the “Single Level of Abstraction Principle”. That is, the first line is written at a much higher level of abstraction than the rest of the lines. I should refactor based on this. Another way to look at the same goal is that the code to write the file is hard to follow.

First I’ll extract the code to construct the name of the report file. Done:

defp report_file_name(module, name), do: "results/#{simplify_module_name(module)}.#{name}.results"

Next, add a function with a nice name to encapsulate the Enum.map. Done:

defp report_single_time(list_of_times, file), do: Enum.map(list_of_times, &IO.puts(file, &1))

Hmm, the name report_single_result doesn’t make sense. It reports from single_bmark result not single run. I renamed it report_single_bmark. In total I have:

defp report_single_bmark({module, name, list_of_times}) do
  filename = report_file_name(module, name)
  File.open(filename, [:write], fn(file) ->
    report_single_time(list_of_times, file)
  end)
end

defp report_file_name(module, name), do: "results/#{simplify_module_name(module)}.#{name}.results"

defp report_single_time(list_of_times, file), do: Enum.map(list_of_times, &IO.puts(file, &1))

Next, I need to clean up

defp simplify_module_name(module) do
  ["Elixir", mod] = Atom.to_string(module) |> String.split(".", parts: 2)
  mod |> String.downcase
end

I don’t even remember what this string manipulation does. Reading it a few times and thinking it removes “Elixir” from teh start of the module name and then downcases. This gives me the most of the name. I introduce a new function with a more intention revealing name and end up with:

defp simplify_module_name(module) do
  strip_elixir_from_module_name(module) |> String.downcase
end

defp strip_elixir_from_module_name(module) do
  ["Elixir", mod] = Atom.to_string(module) |> String.split(".", parts: 2)
  mod
end

bmark_cmp.exs

This code looks pretty good to me.

So then all of the code has been cleaned up.

TODO List

• Clean up the code
• Configurable number of runs
• Use builtin Elixir server module for Bmark.Server?
• Add a readme
• Add a license
• Create a hex package
• Test
• Release

Next Steps

Next week I’ll continue working through the to-do list and will make my way toward release.