Simplifying Integration Tests for Dialyzer

Sorry loyal readers, it has been quite some time since my last post. I thought quitting my job and taking a break would have given me more time to work on Learning Elixir but I ended up taking some time to relax and recover.

In the last post I worked through the details of running Dialyzer on my integration tests in order to type check the my API. The process worked well but required quite a bit of duplicated of code. In this post I would like to try to simplify the process with the hopes of extracting out a framework that can be used to write Dialyzerable integration tests.

I suggest you read the previous post if you haven’t already done so.

What should the integration tests look like?

If I’m going to simplify the test structure then what would I want it to look like?

test/integration/blocking_queue_test.ex would ideally look like a normal ExUnit test file. More or less like what I started with before making the changes to support Dialyzer.
test/integration_test.exs would ideally not exist at all, if it does have to exist then it should have little in it.

That is, I want the support for Dialyzer to require as little overhead and duplication as possible. How can I achieve this?

Elixir Macros

I’ll need to use Elixir Macros to generate the code I need. I can easily imagine generating this code:

# test/integration_test.exs
defmodule BlockingQueue.IntegrationTest do
  use ExUnit.Case
  use ExCheck

  test "BlockingQueue is started with a maximum depth" do
    BlockingQueueTest.test_BlockingQueue_is_started_with_a_maximum_depth
  end

  test "BlockingQueue can push", do: BlockingQueueTest.test_BlockingQueue_can_push

  test "BlockingQueue pop should return the last push" do
    BlockingQueueTest.test_BlockingQueue_pop_should_return_the_last_push
  end

  test "BlockingQueue push and pop should be first in first out" do
    BlockingQueueTest.test_BlockingQueue_push_and_pop_should_be_first_in_first_out
  end

  property "BlockingQueue supports async and blocking pushes and pops" do
    BlockingQueueTest.property_BlockingQueue_supports_async_and_blocking_pushes_and_pops
  end
end

This code is so regular that it should be straightfoward to generate. I should also be able to do something to make these test functions:

def test_BlockingQueue_is_started_with_a_maximum_depth do
  {:ok, _pid} = BlockingQueue.start_link(5)
end

look more like ExUnit tests.

A macro to generate a test shell

First I’ll write a macro to generate these test shells:

test "BlockingQueue is started with a maximum depth" do
  BlockingQueueTest.test_BlockingQueue_is_started_with_a_maximum_depth
end

I’ll include this functionality in a new module just called X within my blocking_queue library. Eventually, I’ll extract this into a separate framework.

I rewrite the test shell as simply:

X.itest BlockingQueueTest, "BlockingQueue is started with a maximum depth"

Then I have to write the X.itest/2 macro:

defmodule X do
  defp name_to_function(name) do
    "test_" <>
    name
    |> String.replace(" ", "_")
    |> String.to_atom
  end

  defmacro itest(module, name) do
    quote do
      test unquote(name) do
        unquote(module).unquote(name_to_function(name))
      end
    end
  end
end

The first part of this is the function name_to_function/1 which converts a name like “Module does stuff” to “test_Module_does_stuff” as an atom.

The macro itest just forms the test code we want using ExUnit.Case.test/2 and name_to_function/1.

We can create a similar macro for ExCheck properties like this:

defmacro iproperty(module, name) do
  quote do
    property unquote(name) do
      unquote(module).unquote(name_to_function("property", name))
    end
  end
end

I had to modify name_to_function/1 slightly to take a prefix so I can use either “test” or “property”. It now looks like this:

defp name_to_function(prefix, name) do
  prefix <> "_" <> name
  |> String.replace(" ", "_")
  |> String.to_atom
end

Of course I made the coresponding change to itest to pass “test”.

With this I can simplify the whole test module as:

defmodule BlockingQueue.IntegrationTest do
  use ExUnit.Case
  use ExCheck
  require X

  X.itest BlockingQueueTest, "BlockingQueue is started with a maximum depth"
  X.itest BlockingQueueTest, "BlockingQueue can push"
  X.itest BlockingQueueTest, "BlockingQueue pop should return the last push"
  X.itest BlockingQueueTest, "BlockingQueue push and pop should be first in first out"
  X.iproperty BlockingQueueTest, "BlockingQueue supports async and blocking pushes and pops"
end

This is much cleaner, though it would be better if I didn’t have to have this module at all.

Clean up the tests themselves

The test code itself is fine. But I should at least have a way to insure that the test names match what I used in the ExUnit test wrappers. As a start I can use the X.name_to_function/2.

To share this code I’ll start by extracting the X module into a new file. For now, called x.ex. Then, I wrote a new macro:

defmacro defitest(name, block) do
  test = name_to_function("test", name)
  quote bind_quoted: binding do
    def unquote(test), do: unquote(block)
  end
end

which is supposed to define the test function with the same name used by the itest. However, the function generated by this macro doesn’t compile. In blocking_queue_test.ex I added an invocation of the macro like this:

defmodule BlockingQueueTest do
  import ExUnit.Assertions
  use ExCheck
  require X

  X.defitest "BlockingQueue is started with a maximum depth" do
    {:ok, _pid} = BlockingQueue.start_link(5)
  end

  #...
end

But the compilation fails like this:

== Compilation error on file test/integration/blocking_queue_test.ex ==
** (CompileError) test/integration/blocking_queue_test.ex:6: invalid syntax in def :test_BlockingQueue_is_started_with_a_maximum_depth
    (elixir) src/elixir_def.erl:44: :elixir_def.store_definition/6
    test/integration/blocking_queue_test.ex:6: (module)
    (stdlib) erl_eval.erl:657: :erl_eval.do_apply/6

It took me quite some time to understand the problem here. For a long time I thought there was something wrong with using an atom as the function name even though this works in other macros. Eventually, I realized I needed this very small change:

@@ -16,7 +16,7 @@ defmodule X do
   defmacro defitest(name, block) do
     test = name_to_function("test", name)
     quote bind_quoted: binding do
-      def unquote(test), do: unquote(block)
+      def unquote(test)(), do: unquote(block)
     end
   end

The empty parentheses for the function are required!

With this I have fully generated a test. I can now convert over the rest of blocking_queue_test.ex. However, there were some issues getting the property test to work. I’m going to skip the property tests for now. That means, I’m left with:

# test/integration_test.exs
defmodule BlockingQueue.IntegrationTest do
  use ExUnit.Case
  use ExCheck
  require X

  X.itest BlockingQueueTest, "BlockingQueue is started with a maximum depth"
  X.itest BlockingQueueTest, "BlockingQueue can push"
  X.itest BlockingQueueTest, "BlockingQueue pop should return the last push"
  X.itest BlockingQueueTest, "BlockingQueue push and pop should be first in first out"
end

# test/integration/blocking_queue_test.ex
defmodule BlockingQueueTest do
  import ExUnit.Assertions
  use ExCheck
  require X

  X.defitest "BlockingQueue is started with a maximum depth" do
    {:ok, _pid} = BlockingQueue.start_link(5)
  end

  X.defitest "BlockingQueue can push" do
    {:ok, pid} = BlockingQueue.start_link(5)
    BlockingQueue.push(pid, "Hi")
  end

  X.defitest "BlockingQueue pop should return the last push" do
    item = "Hi"
    {:ok, pid} = BlockingQueue.start_link(5)
    BlockingQueue.push(pid, item)
    assert item == BlockingQueue.pop(pid)
  end

  X.defitest "BlockingQueue push and pop should be first in first out" do
    {:ok, pid} = BlockingQueue.start_link(5)
    BlockingQueue.push(pid, "Hello")
    BlockingQueue.push(pid, "World")
    assert "Hello" == BlockingQueue.pop(pid)
    assert "World" == BlockingQueue.pop(pid)
  end
end

Next Steps

I would still really like to get rid of test/integration_test.exs as it contains no new content. It’s all information that is already contained in blocking_queue_test.ex.

The next step is to explore ways to generate integration_test.exs from blocking_queue_test.ex. I would like to be able to simplify test/integration_test.exs to:

# test/integration_test.exs
defmodule BlockingQueue.IntegrationTest do
  require X
  something BlockingQueueTest
end

But at this point I’m not sure how to do this. I will continue to research Elixir macros and other techniques that I could use to automate this process. I hope to be able to continue this work in a future post. If you have any ideas or suggestions please leave a comment.

I also need to fix X.defipropert so I can actually generate and Dialyze property tests.

Joseph Kain