Using Dialyzer with Elixir Integration Tests

I have been writing reusable modules, like Blocking Queue which could be used in an appliction. That is, Blocking Queue is not an application on its own. It’s more like a library (to borrow a term form other languages).

I’ve been working on a few projects like this lately and I’ve found that I have a particular workflow. I do two things to try to maintain the quality of the code in these modules

I write tests, mostly integration tests.
I write typespecs and check them with Dialyzer

But, I have found that this workflow is lacking. I really want to be able to run Dialyzer over the integration tests. I want to do this in order to make sure that the tests and the module’s API match up. There are several reasons for this: first it makes sure the tests are correctly, but more importantly it makes sure that the types I’ve chosen for the API are correct or at least usable. If I can’t write tests that conform to the API then the API is too hard to use.

By default Dialyzer doesn’t run on tests because Dialyzer analyzes .beam files and tests are written as .exs scripts which are not compiled to BEAM.

In this post I will explain how I setup my tests so that I can compile then and run Dialyzer on them

Theory

I asked about how to do this on the elixir-lang-talk mailing list. and got a very helpful reply from José Valim:

You can use the elixirc_paths configuration to compile extra directories alongside your code:

https://github.com/phoenixframework/phoenix/blob/699211e95a703250aa5a9711d3cd0cc88e97a7df/installer/templates/new/mix.exs#L27-L28

Then you can define a module with assertions inside your elixirc_paths, for example, test/integration/foo.ex:

 defmodule MyApp.Foo do
   import ExUnit.Assertions
 end

And define a regular test case that calls the functions in the new module, for example, test/my_app/foo_test.exs:

 def MyApp.FooTest do
   use ExUnit.Case
   test "one", do: MyApp.Foo.one
   test "two", do: MyApp.Foo.two
 end

This way you have a module only for tests, that would be compiled like regular code in the test environment, that would integrate with dialyzer, and another module that run its tests. The only note is that you will need to run dialyzer in the test environment:

MIX_ENV=test mix dialyze

In the rest of the post I’ll follow José’s advice and will explain in great detail.

Compiling ExUnit Tests

The first step, is move the integration tests to another directory and rename it .ex instead of .exs. I’ve moved test/blocking_queue_test.exs to test/integeration/blocking_queue_test.ex.

Next, I add the following to mix.exs:

# Specifies which paths to compile per environment
defp elixirc_paths(:test), do: ["lib", "test/integration"]
defp elixirc_paths(_),     do: ["lib"]

This differs slightly from the example José cited because I’m not using Phoenix and without Phoenix the default elixirc_paths is just ["lib"].

I also added this line to the Mix project:

elixirc_paths: elixirc_paths(Mix.env)

I tested this out by running mix test but what I have isn’t enough. The tests in blocking_queue_test.ex can’t use the test macro they need to be regular functions. As regular functions, the tests can’t use strings for their names. After fixing these issue mix test compiles correctly. I ended up with:

# test/integration/blocking_queue_test.ex
defmodule BlockingQueueTest do
  import ExUnit.Assertions
  use ExCheck

  def test_BlockingQueue_is_started_with_a_maximum_depth do
    {:ok, _pid} = BlockingQueue.start_link(5)
  end

  def test_BlockingQueue_can_push do
    {:ok, pid} = BlockingQueue.start_link(5)
    BlockingQueue.push(pid, "Hi")
  end

  def test_BlockingQueue_pop_should_return_the_last_push do
    item = "Hi"
    {:ok, pid} = BlockingQueue.start_link(5)
    BlockingQueue.push(pid, item)
    assert item == BlockingQueue.pop(pid)
  end

  def test_BlockingQueue_push_and_pop_should_be_first_in_first_out do
    {:ok, pid} = BlockingQueue.start_link(5)
    BlockingQueue.push(pid, "Hello")
    BlockingQueue.push(pid, "World")
    assert "Hello" == BlockingQueue.pop(pid)
    assert "World" == BlockingQueue.pop(pid)
  end

  def property_BlockingQueue_supports_async_and_blocking_pushes_and_pops do
    for_all xs in list(int) do
      implies length(xs) > 0 do
        {:ok, pid} = BlockingQueue.start_link(5)
        Task.async(fn ->
          Enum.map(xs, fn x -> BlockingQueue.push(pid, x) end)
        end)

        puller = Task.async(fn ->
          Enum.map(xs, fn _ -> BlockingQueue.pop(pid) end)
        end)

        Task.await(puller) == xs
      end
    end
  end
end

Bringing the tests back to ExUnit

The next step is to add a test file that just calls my integration test functions within ExUnit tests. I ended up with:

# test/integration_test.exs
defmodule BlockingQueue.IntegrationTest do
  use ExUnit.Case
  use ExCheck

  test "BlockingQueue is started with a maximum depth" do
    BlockingQueueTest.test_BlockingQueue_is_started_with_a_maximum_depth
  end

  test "BlockingQueue can push", do: BlockingQueueTest.test_BlockingQueue_can_push

  test "BlockingQueue pop should return the last push" do
    BlockingQueueTest.test_BlockingQueue_pop_should_return_the_last_push
  end

  test "BlockingQueue push and pop should be first in first out" do
    BlockingQueueTest.test_BlockingQueue_push_and_pop_should_be_first_in_first_out
  end

  property "BlockingQueue supports async and blocking pushes and pops" do
    BlockingQueueTest.property_BlockingQueue_supports_async_and_blocking_pushes_and_pops
  end
end

Certainly this the format of each test is regular enough that I could write a macro for this. I’ll put this down as a topic to visit later.

Running Dialyzer on ExUnit Tests

Finally, I can try running Dialyzer (in the test environment) to see how the types line up between my API and the usage in the tests.

$ MIX_ENV=test mix dialyzer
Starting Dialyzer
dialyzer --no_check_plt --plt /Users/jkain/.dialyxir_core_17_1.0.4.plt -Wunmatched_returns -Werror_handling -Wrace_conditions -Wunderspecs /Users/jkain/Documents/Projects/elixir/blocking_queue/_build/test/lib/blocking_queue/ebin
  Proceeding with analysis...
blocking_queue_test.ex:33: Expression produces a value of type #{}, but this value is unmatched
Unknown functions:
  triq_dom:int/0
  triq_dom:list/1
 done in 0m0.76s
done (warnings were emitted)

It works! And it uncovered an error! Though, the error that it uncovered is just a small problem in the test itself. It seems that the types described by my Blocking Queue API and the tests are consistent. But it’s good that I am able to check (and keep them consistent going forward).

I can fix the one error reported by assinging the value to _ like this:

_ = Task.async(fn ->
  Enum.map(xs, fn x -> BlockingQueue.push(pid, x) end)
end)

Next Steps

In this post I’ve followed José Valim’s advice and have been able to run Dialyzer over my integration tests. I’m happy with the results and feel like this will improve my ability to test and write high quality Elixir code.

However, there are some redundancies in the test code itself. The file test/integration_test.exs provides no new information: it just imports external test functions into a file that ExUnit can work with. I would like to experiment with this a bit to see if this workflow could be simplified. The management of the test files and configuration of the test list should be easier to use.

Since I now have the original, recommended process working, next week I can try simplifying and automating it.

Joseph Kain