Custom Types in Ecto

Last week I wrote a small example of using fragment to query overlapping date ranges from an Ecto repo. In this post I’ll continue using the same example but will show how I use custom Ecto types to manage data conversion.

I’ll be using Elixir 1.2.1 and the following dependencies (according to mix.lock):

%{"connection": {:hex, :connection, "1.0.2"},
  "db_connection": {:hex, :db_connection, "0.2.3"},
  "decimal": {:hex, :decimal, "1.1.1"},
  "ecto": {:hex, :ecto, "1.1.3"},
  "poolboy": {:hex, :poolboy, "1.5.1"},
  "postgrex": {:hex, :postgrex, "0.11.0"}}

Date Formats

I’m working on a Phoenix project and the frontend generates dates in the format “DD/MM/YYYY”. This is a common format in my locale. However, my model uses the Ecto.Date type and and won’t cast this format. Consulting the documentation, I see that the cast/1 function supports:

• a binary in the “YYYY-MM-DD” format
• a binary in the “YYYY-MM-DD HH:MM:DD” format (may be separated by T and/or followed by “Z”, as in 2014-04-17T14:00:00Z)
• a binary in the “YYYY-MM-DD HH:MM:DD.USEC” format (may be separated by T and/or followed by “Z”, as in 2014-04-17T14:00:00.030Z)
• a map with “year”, “month” and “day” keys with integer or binaries as values
• a map with :year, :month and :day keys with integer or binaries as values
• a tuple with {year, month, day} as integers or binaries
• an Ecto.Date struct itself

Currently, I have some conversion code to translate the date in “DD/MM/YYYY” format to one of the formats accepted by Ecto.Date.cast/1 but I have the code in my controller. This is the wrong place for such code. I want to build something cleaner.

Example Code

I’ll be building the code for this post based on the code from last week’s post. If you haven’t read last weeks post you may want to. Though the custom type we will develop here will be somewhat orthogonal to the rest of the project. One thing I’ll from the old code is the Ecto setup so I won’t repeat that here.

The github repo for the sample code is here.

A Custom Ecto Type

Given that my application needs to support a single date format, my thought is to write a custom Ecto type that accepts this format in cast/1.

For clarity, in this post I’ll call my new type CustomDate or in full: DateRanges.CustomDate. As part of a real application I would name it based on its function. But in this post its function is to demonstrate a custom type. There are a few other date types involved in the project so I hope this name makes it clear that CustomDate is our custom ecto type.

Looking at the documentation for Ecto.Type, we see that Ecto.Type is a behaviour.

Ecto.Type behaviour

Defines functions and the Ecto.Type behaviour for implementing custom types.

A custom type expects 4 functions to be implemented, all documented and described below.

The documentation goes on to describe the 4 functions:

• type should output the name of the db type
• cast should receive any type and output your custom Ecto type
• load should receive the db type and output your custom Ecto type
• dump should receive your custom Ecto type and output the db type

Now that we have a basic understanding of what’s expected, let’s start implementing.

Ecto Type Cast

We’ll start with cast/1 and to get us going we’ll write a test:

test "it can convert the MM/DD/YYYY format to a date" do
  assert CustomDate.cast("2/9/2016") == {:ok, ??}
end

Hmm, I’m not sure what cast should return here. I know its an :ok tupple but I don’t know what goes in the second field of the tupple. I want the same thing that Ecto.Date would return so let’s write a quick test to understand its return type:

test "it can convert the MM/DD/YYYY format to a date" do
  assert Ecto.Date.cast("2016-09-02") == {:ok, 5}
end

The “5” is clearly wrong. I put it in the test to force a failure - this is the failure:

1) test it can convert the MM/DD/YYYY format to a date (DateRangeTest)
   test/custom_date_test.exs:11
   Assertion with == failed
   code: Ecto.Date.cast("2016-02-09") == {:ok, 5}
   lhs:  {:ok, #Ecto.Date<2016-02-09>}
   rhs:  {:ok, 5}
   stacktrace:
     test/custom_date_test.exs:12

Ah, so Ecto.Date.cast casts to the Ecto.Date struct. I don’t think I want a struct of my own. Instead I think I will wrap Ecto.Date. That is, my cast should also return an Ecto.Date. Now that I’ve figured this out, I can throw away this test. It doesn’t test our code, and we shouldn’t be testing Ecto.Date functionality. But we’ve learned something from our experiment and can put that new knowledge to work and write our own CustomDate test that looks like this:

test "it can convert the MM/DD/YYYY format to a date" do
  assert CustomDate.cast("2/9/2016") == Ecto.Date.cast("2016-02-09")
end

And here’s our first real failure:

1) test it can convert the MM/DD/YYYY format to a date (DateRangeTest)
   test/custom_date_test.exs:7
   ** (UndefinedFunctionError) undefined function CustomDate.cast/1 (module CustomDate is not available)
   stacktrace:
     CustomDate.cast("2/9/2016")
     test/custom_date_test.exs:8

I need to start writing my CustomDate module. Let’s begin with a new file called custom_date.ex:

defmodule DateRanges.CustomDate do
  @behaviour Ecto.Type

  def cast(string) when is_binary(string) do
  end
end

First I’m declaring that this implements the Ecto.Type behavior though we haven’t implemented that behaviour yet.

Second, I’ve written an empty cast/1 function for strings. Now the test fails like this:

1) test it can convert the MM/DD/YYYY format to a date (DateRangeTest)
   test/custom_date_test.exs:7
   Assertion with == failed
   code: CustomDate.cast("2/9/2016") == Ecto.Date.cast("2016-02-09")
   lhs:  nil
   rhs:  {:ok, #Ecto.Date<2016-02-09>}
   stacktrace:
     test/custom_date_test.exs:8

Good, so our function is being called. We just need to return the right value.

This is one possible solution, using a regular expression to parse the date and then converting to a form that Ecto.Date can parse:

defmodule DateRanges.CustomDate do
  @behaviour Ecto.Type

  def cast(string) when is_binary(string) do
    case Regex.run(~r/^([0-9]+)\/([0-9]+)\/([0-9]+)$/, string) do
      [_match, m, d, y] -> Ecto.Date.cast {y, m, d}
      nil -> :error
    end
  end
end

And with this our test passes.

We need to accept a few more formats. For example, if we already have an Ecto.Date we should take it as is:

test "it should accept Ecto.Date" do
  {:ok, ed} = Ecto.Date.cast("2016-02-09")
  assert match?({:ok, _}, CustomDate.cast(ed))
end

And of course the test fails:

1) test it should accept Ecto.Date (CustomDateTest)
   test/custom_date_test.exs:10
   ** (FunctionClauseError) no function clause matching in DateRanges.CustomDate.cast/1
   stacktrace:
     (date_ranges) lib/date_ranges/custom_date.ex:4: DateRanges.CustomDate.cast({:ok, #Ecto.Date<2016-02-09>})
     test/custom_date_test.exs:12

because we need to implement another clause for our cast function:

def cast(%Ecto.Date{} = date), do: Ecto.Date.cast(date)

I decided to call Ecto.Date.cast/1 here. It might not be necessary but why second guess Ecto.Date?

Are there other types we should handle? Well, thinking about it I think I want to support most of the other cases that Ecto.Date supports. That is:

• a map with “year”, “month” and “day” keys with integer or binaries as values
• a map with :year, :month and :day keys with integer or binaries as values
• a tuple with {year, month, day} as integers or binaries

Again, these can be passed through to Ecto.Date.cast/1. Let’s add one more test:

test "it should accept date tuples" do
  assert CustomDate.cast({2016, 2, 9}) == Ecto.Date.cast("2016-02-09")
end

the test fails because we need another clause in our cast function. This implementation will work:

def cast({y, m, d} = date), do: Ecto.Date.cast(date)

The tests passes. But notice that this is almost the same function we wrote to accept an Ecto.Date structure. The only difference is in the accepted pattern. We can combine the two clauses into:

def cast(date), do: Ecto.Date.cast(date)

The whole module looks like this now:

defmodule DateRanges.CustomDate do
  @behaviour Ecto.Type

  def cast(string) when is_binary(string) do
    case Regex.run(~r/^([0-9]+)\/([0-9]+)\/([0-9]+)$/, string) do
      [_match, m, d, y] -> Ecto.Date.cast {y, m, d}
      nil -> :error
    end
  end
  def cast(date), do: Ecto.Date.cast(date)
end

With this change the tests still pass. However, we have expanded our functionality and all of the types accepted by Ecto.Date.cast/1 should work now. But, our cast(string) comes first and will override Ecto.Date’s version.

I could write tests for all the types Ecto.Date supports but, I won’t. I’m leaving this responsibility for these other formats to Ecto and its tests. One nice thing about wrapping Ecto.Date this way is that Ecto could add support for new formats and CustomDate would inherit them automatically.

Dumping Dates

The next step in implementing the Ecto.Type behavior is to write a dump funciton. This function should take our internal type and generate a value appropriate for the database. Since we are wrapping Ecto.Date we should be able to leverage it’s implementation.

But, we start with a test:

test "it should dump dates" do
  {:ok, ed} = Ecto.Date.cast("2016-02-09")
  assert CustomDate.dump(ed) == ??
end

Hmm, again I’m not sure what to expect here. What does Ecto.Date dump? Again, we’ll write a test to experiment:

test "it should dump dates" do
  {:ok, ed} = Ecto.Date.cast("2016-02-09")
  assert Ecto.Date.dump(ed) == 5
end

This test fails and tells us what we should expect:

1) test it should dump dates (CustomDateTest)
   test/custom_date_test.exs:24
   Assertion with == failed
   code: Ecto.Date.dump(ed) == 5
   lhs:  {:ok, {2016, 2, 9}}
   rhs:  5
   stacktrace:
     test/custom_date_test.exs:26

Now we can write our proper test:

test "it should dump dates" do
  {:ok, ed} = Ecto.Date.cast("2016-02-09")
  assert CustomDate.dump(ed) == {:ok, {2016, 2, 9}}
end

The test fails and tells us we need to implement CustomDate.dump. We can do this using delegate like this:

defdelegate dump(x), to: Ecto.Date

And with this change the test passes.

Loading Dates

Now that we have dump/1 implemented we need to implement load/1. This function should take the database type and return an Ecto.Date. Again, we will wrap Ecto.Date’s functionality and we won’t have much to do.

We’ll write a test. Since we’ve just written the dump/1 test we already know what values to expect and won’t need an experimental test to help us out. Our test looks like this:

test "it loads dates" do
  assert CustomDate.load({2016, 2, 9}) == Ecto.Date.cast("2016-02-09")
end

this test fails and tells us to implement the load function. Again, we’ll simply use delegation:

defdelegate load(x), to: Ecto.Date

And our test passes.

Define the type

Along the way the Elixir compiler has been giving me warnings. For example:

lib/date_ranges/custom_date.ex:1: warning: undefined behaviour function dump/1 (for behaviour Ecto.Type)
lib/date_ranges/custom_date.ex:1: warning: undefined behaviour function load/1 (for behaviour Ecto.Type)

There is now one warning left:

lib/date_ranges/custom_date.ex:1: warning: undefined behaviour function type/0 (for behaviour Ecto.Type)

We need to define the type we are using. Now that we have cast/1, dump/1, and load/1 written we have a pretty good idea of what our type is. The type is Ecto.Date. So, we can fix this last warning like this:

def type, do: Ecto.Date

And with this we have a complete Ecto type!

Testing our Ecto.Type

Now that we have this type we need to put it to use and test it. Here’s a test:

test "it can build a changeset" do
  changeset = Ecto.Changeset.cast(%DateRange{},
                %{start: "2/9/2016", end: "5/9/2016"},
                ~w(start end), ~w())
  assert changeset.valid?
end

This test fails:

1) test it can build a changeset (DateRangesTest)
   test/date_ranges_test.exs:10
   Expected truthy, got false
   code: changeset.valid?()
   stacktrace:
     test/date_ranges_test.exs:14

This failure isn’t very enlightening. Let’s change it slightly:

test "it can build a changeset" do
  changeset = Ecto.Changeset.cast(%DateRange{},
                %{start: "2/9/2016", end: "5/9/2016"},
                ~w(start end), ~w())
  assert changeset.errors == []
end

I’ve changed the assert from checking the valid? flag to asserting that there are no errors. This has the affect of printing the errors on failure:

1) test it can build a changeset (DateRangesTest)
   test/date_ranges_test.exs:10
   Assertion with == failed
   code: changeset.errors() == []
   lhs:  [start: "is invalid", end: "is invalid"]
   rhs:  []
   stacktrace:
     test/date_ranges_test.exs:14

Now, this is a little more helpful.

So we are not casting start and end properly. The first problem is that we need to update our schema to use our new type. Let’s make this change:

@@ -3,8 +3,8 @@ defmodule DateRanges.DateRange do
   import Ecto.Query, only: [from: 1, from: 2]

   schema "date_ranges" do
-    field :start, Ecto.Date
-    field :end, Ecto.Date
+    field :start, DateRanges.CustomDate
+    field :end, DateRanges.CustomDate

     timestamps
   end

And with this change the test passes!

Next Steps

For me, the next step will be to integrate this custom type and its tests into my Phoenix based project. For you, I hope you’ll take what we learned here and look to see where you can apply it in your own Ecto and Phoenix based projects.