Learning ElixirJekyll2023-10-01T10:30:53-07:00http://learningelixir.joekain.com/Joseph Kainhttp://learningelixir.joekain.com/joekain@gmail.comhttp://learningelixir.joekain.com/using-gen-stage-to-notify-a-channel2016-10-18T00:00:00-07:002016-10-18T00:00:00-07:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>Last week I <a href="/gen-stage/">learned about <code class="highlighter-rouge">GenStage</code></a>
by reading the <a href="http://elixir-lang.org/blog/2016/07/14/announcing-genstage/">announcement</a>
. It was a good introduction, but now I’m ready to get my hands dirty and write some <code class="highlighter-rouge">GenStage</code> code myself. From the previous post I built up a TODO list of things to try:</p>
<ol>
<li>Watch the ElixirConf 2016 Keynote</li>
<li>Use <code class="highlighter-rouge">GenStage</code> to notify a Phoenix Channel of new events.</li>
<li>Experiment with <code class="highlighter-rouge">Flow</code></li>
<li>Rewrite Domain Scrapper to use <code class="highlighter-rouge">GenStage</code>.</li>
</ol>
<p>I’ve watched the Keynote, so let’s dive into 2.</p>
<h2 id="use-genstage-to-notify-a-phoenix-channel-of-new-events">Use GenStage to Notify a Phoenix Channel of new events.</h2>
<p>This post will follow in the footsteps of <a href="/using-genevent-to-notify-a-channel/">Using GenEvent to Notify a Channel of Updates in Elixir</a>.</p>
<p>In the post using <code class="highlighter-rouge">GenEvent</code>, we had a channel that we were using to broadcast
changes to a model to all web clients. But, we wanted to decouple the controller
that was performing updates to the model and the channel. The thought being that
we would use an observer pattern to communicate changes. There could be many
observers to the changes in addition to the channel and the controller shouldn’t
have to be aware of any of them.</p>
<p>I have a
<a href="https://github.com/joekain/play_channel">play channel project in github</a>. I
went back and created a branch called pre-gen-event and will begin work from
there.</p>
<p>First, I need to insure that I can still build everything. There are some warnings in the deps. These are because the deps are old but I’ve upgraded to Elixir 1.3.2. I don’t want to update Phoenix yet so I’m going to live with the warnings for now.</p>
<p>There is one warning in my code:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>warning: variable payload is unused
web/channels/toy_channel.ex:4
</code></pre>
</div>
<p>This is easy enough to fix by renaming <code class="highlighter-rouge">payload</code> to <code class="highlighter-rouge">_payload</code> in the argument list to <code class="highlighter-rouge">join/3</code>.</p>
<p>At this point the project compiles and all the tests pass.</p>
<p>To get started with our GenStage changes, we’ll have to add <code class="highlighter-rouge">Experimental.GenStage</code> to our dependencies in mix.exs by adding a line like this to <code class="highlighter-rouge">deps</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="p">{</span><span class="ss">:gen_stage</span><span class="p">,</span> <span class="sd">"</span><span class="s2">~> 0.5"</span><span class="p">}]</span></code></pre></figure>
<p>Then, I’ll add <code class="highlighter-rouge">alias Experimental.GenStage</code> to the top of toy_channel.ex and
I’ll add <code class="highlighter-rouge">use GenStage</code> just after <code class="highlighter-rouge">use PlayChannel.Web, :channel</code> at the
begining of the module.</p>
<p>These changes trigger a few warnings:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>warning: undefined behaviour function init/1 (for behaviour Experimental.GenStage)
web/channels/toy_channel.ex:3
warning: conflicting behaviours - callback code_change/3 required by both ''Elixir.Phoenix.Channel'' and ''Elixir.Experimental.GenStage'' (line 3)
web/channels/toy_channel.ex:3
warning: conflicting behaviours - callback handle_info/2 required by both ''Elixir.Phoenix.Channel'' and ''Elixir.Experimental.GenStage'' (line 3)
web/channels/toy_channel.ex:3
warning: conflicting behaviours - callback terminate/2 required by both ''Elixir.Phoenix.Channel'' and ''Elixir.Experimental.GenStage'' (line 3)
web/channels/toy_channel.ex:3
</code></pre>
</div>
<p>The tests still pass but I’m worried about these warnings. There are two types. Here’s the first:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>warning: undefined behaviour function init/1 (for behaviour Experimental.GenStage)
</code></pre>
</div>
<p>This is a good warning and we can deal with it. I just haven’t implemented GenStage’s required <code class="highlighter-rouge">init/1</code> function. Once we add the function it should address the warning. We’re left with:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>warning: conflicting behaviours - callback code_change/3 required by both ''Elixir.Phoenix.Channel'' and ''Elixir.Experimental.GenStage'' (line 3)
warning: conflicting behaviours - callback handle_info/2 required by both ''Elixir.Phoenix.Channel'' and ''Elixir.Experimental.GenStage'' (line 3)
web/channels/toy_channel.ex:3
warning: conflicting behaviours - callback terminate/2 required by both ''Elixir.Phoenix.Channel'' and ''Elixir.Experimental.GenStage'' (line 3)
web/channels/toy_channel.ex:3
</code></pre>
</div>
<p>For these three, we had similar warning when adding GenEvent to the channel. And again, these are the OTP compliant callbacks and will have the same purpose for both <code class="highlighter-rouge">GenStage</code> and <code class="highlighter-rouge">Phoenix.Channel</code>. So I think we’re ok to proceed here.</p>
<p>Well, with that out of the way let’s go ahead and implement the <code class="highlighter-rouge">init/1</code> function. To do this, we need to decide if <code class="highlighter-rouge">PlayChannel.ToyChannel</code> is a <code class="highlighter-rouge">GenStage</code> producer or consumer.</p>
<p>Our goal here is to use GenStage “to Notify a Channel of Updates”. That means that</p>
<ul>
<li>a change will original in the controller</li>
<li>there is a notification center to broadcast an event about the change</li>
<li><code class="highlighter-rouge">PlayChannel.ToyChannel</code> receives the event and sends the payload over the channel.</li>
</ul>
<p>This means the notification center will be the event producer and <code class="highlighter-rouge">PlayChannel.ToyChannel</code> will be the consumer.</p>
<p>Now we can write the <code class="highlighter-rouge">init/1</code> function and intialize as a consumer. Here’s the function we need:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"> <span class="c1"># GenStage callbacks</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:consumer</span><span class="p">,</span> <span class="no">nil</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>Our <code class="highlighter-rouge">init/1</code> needs no arguments and no state. It only need to return that it will behave as a <code class="highlighter-rouge">GenStage</code> <code class="highlighter-rouge">:consumer</code>.</p>
<p>At this point the tests still pass. However, we need to add a <code class="highlighter-rouge">handle_events/3</code> function in order to receive events from our notification center. I think this should do it:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">handle_events</span><span class="p">(</span><span class="n">events</span><span class="p">,</span> <span class="n">_from</span><span class="p">,</span> <span class="n">_state</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">each</span> <span class="n">events</span><span class="p">,</span> <span class="k">fn</span>
<span class="p">{</span><span class="ss">:update</span><span class="p">,</span> <span class="n">toy</span><span class="p">}</span> <span class="o">-></span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">ToyChannel</span><span class="o">.</span><span class="n">broadcast_change</span><span class="p">(</span><span class="n">toy</span><span class="p">)</span>
<span class="k">end</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="p">[],</span> <span class="no">nil</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>This is similar to what we wrote for <code class="highlighter-rouge">GenEvent</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">handle_event</span><span class="p">({</span><span class="ss">:update</span><span class="p">,</span> <span class="n">toy</span><span class="p">},</span> <span class="n">_</span><span class="p">)</span> <span class="k">do</span>
<span class="no">PlayChannel</span><span class="o">.</span><span class="no">ToyChannel</span><span class="o">.</span><span class="n">broadcast_change</span><span class="p">(</span><span class="n">toy</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">nil</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>Except that <code class="highlighter-rouge">GenStage.handle_events/3</code> accepts a list of events. To process the list we use <code class="highlighter-rouge">Enum.each</code> and broadcast each update on the channel. <code class="highlighter-rouge">handle_events</code> returns no events (it’s a consumer) and needs no state.</p>
<h2 id="build-the-genstage-producer">Build the GenStage Producer</h2>
<p>Next, we need to create the producer. We’ll call it
<code class="highlighter-rouge">PlayChannel.Toy.UpdateEventHandler</code> (this is the same name we used in the
GenEvent version). We’ll start with</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span>
<span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandler</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenStage</span>
<span class="k">end</span></code></pre></figure>
<p>There’s actually a great example of what we need in the <a href="http://elixir-lang.org/blog/2016/07/14/announcing-genstage/">GenStage annoucement</a></p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span>
<span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandler</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenStage</span>
<span class="k">def</span> <span class="n">start_link</span><span class="p">()</span> <span class="k">do</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="ss">name:</span> <span class="bp">__MODULE__</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">sync_notify</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">timeout</span> <span class="p">\\</span> <span class="m">5000</span><span class="p">)</span> <span class="k">do</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="p">{</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">timeout</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1">## Callbacks</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="ss">:ok</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="p">{</span><span class="ss">:queue</span><span class="o">.</span><span class="n">new</span><span class="p">,</span> <span class="m">0</span><span class="p">},</span> <span class="ss">dispatcher:</span> <span class="no">GenStage</span><span class="o">.</span><span class="no">BroadcastDispatcher</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_call</span><span class="p">({</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">from</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">})</span> <span class="k">do</span>
<span class="n">dispatch_events</span><span class="p">(</span><span class="ss">:queue</span><span class="o">.</span><span class="ow">in</span><span class="p">({</span><span class="n">from</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">queue</span><span class="p">),</span> <span class="n">demand</span><span class="p">,</span> <span class="p">[])</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_demand</span><span class="p">(</span><span class="n">incoming_demand</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">})</span> <span class="k">do</span>
<span class="n">dispatch_events</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">incoming_demand</span> <span class="o">+</span> <span class="n">demand</span><span class="p">,</span> <span class="p">[])</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">dispatch_events</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">,</span> <span class="n">events</span><span class="p">)</span> <span class="k">do</span>
<span class="n">with</span> <span class="n">d</span> <span class="ow">when</span> <span class="n">d</span> <span class="o">></span> <span class="m">0</span> <span class="o"><-</span> <span class="n">demand</span><span class="p">,</span>
<span class="p">{</span><span class="n">item</span><span class="p">,</span> <span class="n">queue</span><span class="p">}</span> <span class="o">=</span> <span class="ss">:queue</span><span class="o">.</span><span class="n">out</span><span class="p">(</span><span class="n">queue</span><span class="p">),</span>
<span class="p">{</span><span class="ss">:value</span><span class="p">,</span> <span class="p">{</span><span class="n">from</span><span class="p">,</span> <span class="n">event</span><span class="p">}}</span> <span class="o"><-</span> <span class="n">item</span> <span class="k">do</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">reply</span><span class="p">(</span><span class="n">from</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">)</span>
<span class="n">dispatch_events</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span> <span class="o">-</span> <span class="m">1</span><span class="p">,</span> <span class="p">[</span><span class="n">event</span> <span class="o">|</span> <span class="n">events</span><span class="p">])</span>
<span class="k">else</span>
<span class="n">_</span> <span class="o">-></span> <span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="no">Enum</span><span class="o">.</span><span class="n">reverse</span><span class="p">(</span><span class="n">events</span><span class="p">),</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">}}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Let’s go over this.</p>
<p>First there is the <code class="highlighter-rouge">start_link</code> function. This will be used to start and name the GenStage.</p>
<p>Next, <code class="highlighter-rouge">sync_notify</code>. This is the function that our controller will call to notify observers that an update has been made. It uses <code class="highlighter-rouge">GenStage.call</code> to send the update event to the GenStage server.</p>
<p>Next come the callbacks, the first is <code class="highlighter-rouge">init</code>. This initializes the GenStage as a producer and sets a custom dispatch. <code class="highlighter-rouge">GenStage.BroadcastDispatcher</code> is describe in the documentation as:</p>
<blockquote>
<p>A dispatcher that accumulates demand from all consumers before broadcasting events to all of them</p>
</blockquote>
<p>This is the behavior we need for an notification center. Finally, we set up a state for the stage. Out state is a queue, and a demand. The initial queue is created with <code class="highlighter-rouge">:queue.new</code>. This queue will hold events waiting to be broadcast. The initial demand is 0.</p>
<p>At the end of the module is a function, <code class="highlighter-rouge">dispatch_events</code> which is used to implement both <code class="highlighter-rouge">handle_call</code> and <code class="highlighter-rouge">handle_demand</code>.</p>
<p>Let’s think first about <code class="highlighter-rouge">handle_demand</code>. There are two cases to consider
First, there are events in the queue ready to be returned. In this case we can just return them in <code class="highlighter-rouge">handle_demand</code>. Second, the queue is empty. In this case we can’t return a result to the consumer, we have to defer it’s return until there is an event.</p>
<p>The next callback is <code class="highlighter-rouge">handle_call/3</code> This is just like <code class="highlighter-rouge">GenServer</code>’s <code class="highlighter-rouge">handle_call</code> and will receive the messages sent by our <code class="highlighter-rouge">sync_notify</code> function.</p>
<p>Next is <code class="highlighter-rouge">handle_demand/2</code> which is the GenStage callback for a producer to provide more events. This calls <code class="highlighter-rouge">dispatch_events</code> which is an interesting function. It recursively calls itself and for each element in queue up to the demand. For each recursive call it calls <code class="highlighter-rouge">GenStage.reply/2</code>. This is the reply back to the <code class="highlighter-rouge">call</code> for <code class="highlighter-rouge"><span class="p">{</span><span class="err">:notfy</span><span class="w"> </span><span class="err">event</span><span class="p">}</span></code></p>
<p>I’m actually having a hard time understanding and explaining this code. I think I’ll try writing my own version.</p>
<h2 id="building-a-genstage-producer-for-event-notification">Building a GenStage producer for Event Notification</h2>
<p>I want to TDD a solution for <code class="highlighter-rouge">UpdateEventHandler</code>. I had one false start trying to write this up and ended up with tests like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span>
<span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandlerTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span>
<span class="n">alias</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandler</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">it can init"</span> <span class="k">do</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="no">nil</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">it is a producer"</span> <span class="k">do</span>
<span class="n">result</span> <span class="o">=</span> <span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="no">nil</span><span class="p">)</span>
<span class="n">assert</span> <span class="n">elem</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="m">0</span><span class="p">)</span> <span class="o">==</span> <span class="ss">:producer</span>
<span class="k">end</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">it should start_link"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">stage</span><span class="p">}</span> <span class="o">=</span> <span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">start_link</span><span class="p">()</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">stage</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">it should accept events"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">stage</span><span class="p">}</span> <span class="o">=</span> <span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">start_link</span><span class="p">()</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">notify</span><span class="p">(</span><span class="n">stage</span><span class="p">,</span> <span class="sd">"</span><span class="s2">Hi"</span><span class="p">)</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">stage</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">it should broadcast the events"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="no">false</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>As I wrote more and more tests, I started to think that this is too tied to my implementation. Yes, I want to use <code class="highlighter-rouge">GenStage</code>. And in fact I’m writing a blog post about <code class="highlighter-rouge">GenStage</code>. But should my tests really require that? The fact that I’ll use <code class="highlighter-rouge">GenStage</code> is a implementation detail and the tests should really only test the externally visible behavior, right? I’m throwing these tests away and starting over. I’ll focus on the behaviors.</p>
<p>That said, I do have a <code class="highlighter-rouge">GenStage</code> in <code class="highlighter-rouge">PlayChannel.ToyChannel</code> (the observer) and it will need to subscribe the the <code class="highlighter-rouge">UpdateEventHandler</code>. But that’s the only <code class="highlighter-rouge">GenStage</code> detail that I should depend on.</p>
<p>I’ll start with this test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it should start_link and stop"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">pid</span><span class="p">}</span> <span class="o">=</span> <span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">start_link</span><span class="p">()</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>This is adapated from test tests I had before, but I’ve created a new <code class="highlighter-rouge">UpdateEventHandler.stop/1</code> interface. This is nice, it abstracts me away from <code class="highlighter-rouge">GenStage.stop/1</code>.</p>
<p>Here’s an implementation that passes the tests:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span>
<span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandler</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenStage</span>
<span class="k">def</span> <span class="n">start_link</span><span class="p">(),</span> <span class="k">do</span><span class="p">:</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span>
<span class="k">def</span> <span class="n">stop</span><span class="p">(</span><span class="n">stage</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">stage</span><span class="p">)</span>
<span class="c1"># Callbacks</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="no">nil</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>Now, I could have written a simpler implementation that passed the tests. But I’m going to go ahead and write this as a <code class="highlighter-rouge">GenStage</code> because I know I’ll need to.</p>
<p>The next test I’ll add is:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it should accept events"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">pid</span><span class="p">}</span> <span class="o">=</span> <span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">start_link</span><span class="p">()</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">notify</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="sd">"</span><span class="s2">Hi"</span><span class="p">)</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>which requires that I have a <code class="highlighter-rouge">notify</code> function. This is part of the behavior I want in my module. I want callers to be able to notify all observers of an event.</p>
<p>Here’s a very simple implementation that passes:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">notify</span><span class="p">(</span><span class="n">handler</span><span class="p">,</span> <span class="n">event</span><span class="p">)</span> <span class="k">do</span>
<span class="k">end</span></code></pre></figure>
<p>I should also note that I get a few warnings when compiling:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>warning: variable event is unused
lib/play_channel/toy/update_event_handler.ex:9
warning: variable handler is unused
lib/play_channel/toy/update_event_handler.ex:9
</code></pre>
</div>
<p>I could easily fix these, but I keep them as a reminder that <code class="highlighter-rouge">notify</code> is imcomplete, I’m not actually notifying anything yet.</p>
<p>The next behavior that I want to test for is that I can subscribe a <code class="highlighter-rouge">GenStage</code> consumer to the UpdateEventHandler. This test requires that I setup such a consumer. Here’s my shot at it:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">TestObserver</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenStage</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:consumer</span><span class="p">,</span> <span class="no">nil</span><span class="p">}</span>
<span class="k">def</span> <span class="n">handle_events</span><span class="p">(</span><span class="n">events</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="p">[],</span> <span class="n">state</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">an observer can subscribe to it"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">observer</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">TestObserver</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">updater</span><span class="p">}</span> <span class="o">=</span> <span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">start_link</span><span class="p">()</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">observer</span><span class="p">,</span> <span class="ss">to:</span> <span class="n">updater</span><span class="p">)</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">updater</span><span class="p">)</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">observer</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>Adding <code class="highlighter-rouge">TestObserver</code> is kind of a lot of code to write for a test. But I’ve tried to keep it as simple as possible. When I run this test I get:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test an observer can subscribe to it (PlayChannel.Toy.UpdateEventHandlerTest)
test/update_event_handler_test.exs:32
** (EXIT from #PID<0.256.0>) an exception was raised:
** (UndefinedFunctionError) function PlayChannel.Toy.UpdateEventHandler.handle_demand/2 is undefined or private.
</code></pre>
</div>
<p>This means that I need to add the <code class="highlighter-rouge">handle_demand/2</code> callback. Let’s do it:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"> <span class="k">def</span> <span class="n">handle_demand</span><span class="p">(</span><span class="n">demand</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="p">[],</span> <span class="n">state</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>This doesn’t do anything yet. When I run the test it still fails, now with this error:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test an observer can subscribe to it (PlayChannel.Toy.UpdateEventHandlerTest)
test/update_event_handler_test.exs:32
** (exit) no process
stacktrace:
(stdlib) proc_lib.erl:794: :proc_lib.stop/3
test/update_event_handler_test.exs:38: (test)
</code></pre>
</div>
<p>Line 38, in the test, coresponds to the call to <code class="highlighter-rouge">GenStage.stop(observer)</code>. I guess that when I stop <code class="highlighter-rouge">updater</code> the subscribed consumers must also shutdown. So I can just remove the call.</p>
<p>With this change the test passes.</p>
<p>Next, I want to write a test that calls notify on <code class="highlighter-rouge">updater</code> then verifies that <code class="highlighter-rouge">observer</code> sees the event. To do this I’m going to need to add something to <code class="highlighter-rouge">TestObserer</code> to allow me to verify what it observes. I’ll do this before writing any new tests. I think this should do it:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"> defmodule TestObserver do
use GenStage
<span class="gd">- def init(_), do: {:consumer, nil}
</span><span class="gi">+ def init(_), do: {:consumer, []}
</span> def handle_events(events, _, state) do
<span class="gd">- {:noreply, [], state}
</span><span class="gi">+ {:noreply, [], state ++ events}
</span> end
<span class="gi">+
+ def get(stage), do: GenStage.call(stage, :get)
+ def handle_call(:get, _, state), do: {:reply, state, [], state}
</span><span class="err"> end</span></code></pre></figure>
<p>Here I’ve added an interface <code class="highlighter-rouge">get/1</code> which is procesed by <code class="highlighter-rouge">handle_call/3</code>. The get functionality returns the state of the <code class="highlighter-rouge">TestObserver</code>. I’ve also updated <code class="highlighter-rouge">handle_events</code> to just accumulate events into the state. That is, <code class="highlighter-rouge">TestObserver</code> collects its events and returns them when you call <code class="highlighter-rouge">get</code>.</p>
<p>After making this change I reran the existing tests to make sure I didn’t break <code class="highlighter-rouge">TestObserver</code>. All was good.</p>
<p>Next, I wrote this test which uses the new <code class="highlighter-rouge">get</code> function:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">a subscribed observer is notified of all events"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">observer</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">TestObserver</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">updater</span><span class="p">}</span> <span class="o">=</span> <span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">start_link</span><span class="p">()</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">observer</span><span class="p">,</span> <span class="ss">to:</span> <span class="n">updater</span><span class="p">)</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">notify</span><span class="p">(</span><span class="n">updater</span><span class="p">,</span> <span class="sd">"</span><span class="s2">Hi"</span><span class="p">)</span>
<span class="n">assert</span> <span class="no">TestObserver</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">observer</span><span class="p">)</span> <span class="o">==</span> <span class="p">[</span><span class="sd">"</span><span class="s2">Hi"</span><span class="p">]</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">updater</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>This is very similar to the previous test except that we’ve added the assert:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">assert</span> <span class="no">TestObserver</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">observer</span><span class="p">)</span> <span class="o">==</span> <span class="p">[</span><span class="sd">"</span><span class="s2">Hi"</span><span class="p">]</span></code></pre></figure>
<p>This uses the new <code class="highlighter-rouge">get</code> function to retrieve any observed events. And we expect to get the single event “Hi” because that’s the event we notified to the updater.</p>
<p>We can make this new test pass by updating <code class="highlighter-rouge">handle_call</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">handle_call</span><span class="p">({</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">_from</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="p">[</span><span class="n">event</span><span class="p">],</span> <span class="n">state</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>This version declares <code class="highlighter-rouge">[event]</code> as available events, ready to be dispatched. The <code class="highlighter-rouge">GenStage</code> logic will see to it that these events are stored in an internal buffer and dispatched when there is demand.</p>
<p>After writing this I found that the documentation has an example that uses almost the same exact code as I have just written! And it goes on to describe the reason for the more complex version.</p>
<p>The motivation for a more complex implementation is that unserved demand is being held in <code class="highlighter-rouge">GenStage</code>’s internal buffer. Unserved demand refers to events that have arrived through the <code class="highlighter-rouge">notify</code> function but have not yet been delivered to subscribers through the <code class="highlighter-rouge">handle_demand</code> function. If this internal buffer becomes too full a log message will be emitted. A better solution is to manage a queue within our code to hold all the pending events.</p>
<p>Let’s try adding a queue to our version. There’s no test that should motivate this change as the externally visible behavior stays the same.</p>
<p>Our <code class="highlighter-rouge">UpdateEventHandler</code> will need state. Let’s think for a minute what this state is:</p>
<ul>
<li>A queue to hold incoming events. This queue fills up when the rate of incoming events exceeds the demand.</li>
<li>Unfilled demand. This can be a simple counter that records demand that we haven’t been able to fill because the demand exceeds the suppy of events.</li>
</ul>
<p>Let’s put this together. First, in <code class="highlighter-rouge">init</code> we have to intialize our state. We need an empty queue and 0 for outstanding demand.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="p">{</span><span class="ss">:queue</span><span class="o">.</span><span class="n">new</span><span class="p">,</span> <span class="m">0</span><span class="p">}}</span></code></pre></figure>
<p>With this change the tests still pass.</p>
<p>Now, in <code class="highlighter-rouge">handle_call</code> we accept new events. There are two cases to deal with</p>
<ol>
<li>There is no outstanding demand, so we put the new event in our queue.</li>
<li>There is outstanding demand, so we forward the new event on immediately.</li>
</ol>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">handle_call</span><span class="p">({</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">_from</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="m">0</span><span class="p">})</span> <span class="k">do</span>
<span class="n">queue</span> <span class="o">=</span> <span class="ss">:queue</span><span class="o">.</span><span class="ow">in</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">queue</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="p">[],</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="m">0</span><span class="p">}}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_call</span><span class="p">({</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">_from</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">})</span> <span class="k">do</span>
<span class="n">queue</span> <span class="o">=</span> <span class="ss">:queue</span><span class="o">.</span><span class="ow">in</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">queue</span><span class="p">)</span>
<span class="p">{</span> <span class="p">{</span><span class="ss">:value</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">queue</span><span class="p">}</span> <span class="o">=</span> <span class="ss">:queue</span><span class="o">.</span><span class="n">out</span><span class="p">(</span><span class="n">queue</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="p">[</span><span class="n">event</span><span class="p">],</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span> <span class="o">-</span> <span class="m">1</span><span class="p">}}</span>
<span class="k">end</span></code></pre></figure>
<p>Here, we handle the two cases using two separate function clauses. The first, handles the case where there is no outstanding demand (demand is 0).</p>
<p>The second function clause handles the case where there is demand. It inserts the event at the rear of the queue and then pops off an event from the head of the queue and returns it.</p>
<p>Do I need to handle the case that <code class="highlighter-rouge">demand > 1</code> and the queue is not empty in <code class="highlighter-rouge">handle_call</code>? If so, it means that I should return more than one event. But, if there is demand and there were events waiting why were they not already dispatched? This is something to look into a little later.</p>
<p>Also, our tests fail after this change:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test a subscribed observer is notified of all events (PlayChannel.Toy.UpdateEventHandlerTest)
test/update_event_handler_test.exs:39
Assertion with == failed
code: TestObserver.get(observer) == ["Hi"]
lhs: []
rhs: ["Hi"]
stacktrace:
test/update_event_handler_test.exs:46: (test)
</code></pre>
</div>
<p>Ok, this failure makes sense because we’re never increasing the demand stored in our state. So we always use the first case (demand is 0) and never return the events.</p>
<p>To fix this, we need to update <code class="highlighter-rouge">handle_demand</code>. Again, there are two cases.</p>
<ol>
<li>There are waiting events in the queue. In this case satisfy the demand from the queue.</li>
<li>There are no waiting events in the queue. In this case save the demand in the state.</li>
</ol>
<p>Here’s an implementation</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">handle_demand</span><span class="p">(</span><span class="n">demand</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">old_demand</span><span class="p">})</span> <span class="k">do</span>
<span class="p">{</span><span class="n">events</span><span class="p">,</span> <span class="n">new_queue</span><span class="p">,</span> <span class="n">new_demand</span><span class="p">}</span> <span class="o">=</span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span> <span class="o">+</span> <span class="n">old_demand</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">events</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">new_demand</span><span class="p">}}</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">count</span><span class="p">,</span> <span class="n">items</span> <span class="p">\\</span> <span class="p">[])</span>
<span class="k">defp</span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="m">0</span><span class="p">,</span> <span class="n">items</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="n">items</span><span class="p">,</span> <span class="n">queue</span><span class="p">,</span> <span class="m">0</span><span class="p">}</span>
<span class="k">defp</span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">count</span><span class="p">,</span> <span class="n">items</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="ss">:queue</span><span class="o">.</span><span class="n">out</span><span class="p">(</span><span class="n">queue</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span> <span class="p">{</span><span class="ss">:value</span><span class="p">,</span> <span class="n">item</span><span class="p">},</span> <span class="n">queue</span><span class="p">}</span> <span class="o">-></span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">count</span> <span class="o">-</span> <span class="m">1</span><span class="p">,</span> <span class="n">items</span> <span class="o">++</span> <span class="p">[</span><span class="n">item</span><span class="p">])</span>
<span class="p">{</span><span class="ss">:empty</span><span class="p">,</span> <span class="n">queue</span><span class="p">}</span> <span class="o">-></span> <span class="p">{</span><span class="n">items</span><span class="p">,</span> <span class="n">queue</span><span class="p">,</span> <span class="n">count</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>I’ve introduced the function <code class="highlighter-rouge">take</code> to take multiple items out of the queue. It’s a simple recursive function though it would be nice to have tests for it. It returns a tuple of the taken events, a new queue with the remaining events, and the remaining demand.</p>
<p>With this change the tests (the ones we already had) pass.</p>
<p>An interesting thing to note is that we could apply <code class="highlighter-rouge">take</code> in <code class="highlighter-rouge">handle_call</code> to deal with the case of <code class="highlighter-rouge">demand > 1</code> and queue not empty that I asked about earlier. It would look something like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">handle_call</span><span class="p">({</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">_from</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">})</span> <span class="k">do</span>
<span class="n">queue</span> <span class="o">=</span> <span class="ss">:queue</span><span class="o">.</span><span class="ow">in</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">queue</span><span class="p">)</span>
<span class="p">{</span><span class="n">events</span><span class="p">,</span> <span class="n">new_queue</span><span class="p">,</span> <span class="n">new_demand</span><span class="p">}</span> <span class="o">=</span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="n">events</span><span class="p">,</span> <span class="p">{</span><span class="n">new_queue</span><span class="p">,</span> <span class="n">new_demand</span><span class="p">}}</span>
<span class="k">end</span></code></pre></figure>
<p>I guess there is no reason not to use this version.</p>
<p>Here’s the whole module:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span>
<span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandler</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenStage</span>
<span class="k">def</span> <span class="n">start_link</span><span class="p">(),</span> <span class="k">do</span><span class="p">:</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span>
<span class="k">def</span> <span class="n">stop</span><span class="p">(</span><span class="n">stage</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">stage</span><span class="p">)</span>
<span class="k">def</span> <span class="n">notify</span><span class="p">(</span><span class="n">handler</span><span class="p">,</span> <span class="n">event</span><span class="p">)</span> <span class="k">do</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">handler</span><span class="p">,</span> <span class="p">{</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">})</span>
<span class="k">end</span>
<span class="c1"># Callbacks</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="p">{</span><span class="ss">:queue</span><span class="o">.</span><span class="n">new</span><span class="p">,</span> <span class="m">0</span><span class="p">}}</span>
<span class="k">def</span> <span class="n">handle_demand</span><span class="p">(</span><span class="n">demand</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">old_demand</span><span class="p">})</span> <span class="k">do</span>
<span class="p">{</span><span class="n">events</span><span class="p">,</span> <span class="n">new_queue</span><span class="p">,</span> <span class="n">new_demand</span><span class="p">}</span> <span class="o">=</span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span> <span class="o">+</span> <span class="n">old_demand</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">events</span><span class="p">,</span> <span class="p">{</span><span class="n">new_queue</span><span class="p">,</span> <span class="n">new_demand</span><span class="p">}}</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">count</span><span class="p">,</span> <span class="n">items</span> <span class="p">\\</span> <span class="p">[])</span>
<span class="k">defp</span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="m">0</span><span class="p">,</span> <span class="n">items</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="n">items</span><span class="p">,</span> <span class="n">queue</span><span class="p">,</span> <span class="m">0</span><span class="p">}</span>
<span class="k">defp</span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">count</span><span class="p">,</span> <span class="n">items</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="ss">:queue</span><span class="o">.</span><span class="n">out</span><span class="p">(</span><span class="n">queue</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span> <span class="p">{</span><span class="ss">:value</span><span class="p">,</span> <span class="n">item</span><span class="p">},</span> <span class="n">queue</span><span class="p">}</span> <span class="o">-></span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">count</span> <span class="o">-</span> <span class="m">1</span><span class="p">,</span> <span class="n">items</span> <span class="o">++</span> <span class="p">[</span><span class="n">item</span><span class="p">])</span>
<span class="p">{</span><span class="ss">:empty</span><span class="p">,</span> <span class="n">queue</span><span class="p">}</span> <span class="o">-></span> <span class="p">{</span><span class="n">items</span><span class="p">,</span> <span class="n">queue</span><span class="p">,</span> <span class="n">count</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_call</span><span class="p">({</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">_from</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">})</span> <span class="k">do</span>
<span class="n">queue</span> <span class="o">=</span> <span class="ss">:queue</span><span class="o">.</span><span class="ow">in</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">queue</span><span class="p">)</span>
<span class="p">{</span><span class="n">events</span><span class="p">,</span> <span class="n">new_queue</span><span class="p">,</span> <span class="n">new_demand</span><span class="p">}</span> <span class="o">=</span> <span class="n">take</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="n">events</span><span class="p">,</span> <span class="p">{</span><span class="n">new_queue</span><span class="p">,</span> <span class="n">new_demand</span><span class="p">}}</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>This is now pretty close to <code class="highlighter-rouge">QueueBroadcaster</code> in the <code class="highlighter-rouge">GenStage</code> docs. The difference is that I opted for the abstraction of <code class="highlighter-rouge">take</code> which is inspired by <code class="highlighter-rouge">Enum.take/2</code>.</p>
<p><code class="highlighter-rouge">QueueBroadcaster</code> does the same thing but combines extracting items from the queue with building up the return values for <code class="highlighter-rouge">handle_demand</code> and <code class="highlighter-rouge">handle_call</code> in the <code class="highlighter-rouge">dispatch_events</code> function.</p>
<p>Conceptually, I find my version a little easier to read because <code class="highlighter-rouge">handle_demand</code> and <code class="highlighter-rouge">handle_call</code> build their own return values. This way <code class="highlighter-rouge">take</code> just focuses on the single tasks of extracting items from the queue.</p>
<p>That said, there are a couple of things that <code class="highlighter-rouge">QueueBroadcaster</code> does better than <code class="highlighter-rouge">UpdateEventHandler</code></p>
<ol>
<li>
<p><code class="highlighter-rouge">dispatch_events</code> has an optimization in building up the lists. It uses <code class="highlighter-rouge">[event | events]</code> where <code class="highlighter-rouge">take</code> has <code class="highlighter-rouge">items ++ [item]</code>. Inserting at the head of the list is more efficient. <code class="highlighter-rouge">dispatch_events</code> then reverse the list before returning the final result.</p>
</li>
<li>
<p><code class="highlighter-rouge">dispatch_events</code> uses <code class="highlighter-rouge">:noreply</code> in <code class="highlighter-rouge">handle_call</code> if the value isn’t being sent to the observers immediately. Then, when the event is finally consumed it uses <code class="highlighter-rouge">GenStage.reply/2</code> to send the reply. This keeps the caller from going on until the notification is delivered.</p>
</li>
</ol>
<p>I think point number 2 is really important and I need that functionality as well. And I think this is what necessitates <code class="highlighter-rouge">dispatch_events</code> being as complex as it is.</p>
<p>Give this, I think it makes sense to use the <code class="highlighter-rouge">QueueBroadcaster</code> implementation. But working through this implementation has given me a much better understanding of what’s going on.</p>
<p>Also, there is one more thing we should have done with <code class="highlighter-rouge">UpdateEventHandler</code>. We needed to write a test for multiple subscribers. And this would have revealed that we needed to change our init function to use the <code class="highlighter-rouge">BroadcastDispatcher</code> as <code class="highlighter-rouge">QueueBroadcaster</code> does. This would like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="p">{</span><span class="ss">:queue</span><span class="o">.</span><span class="n">new</span><span class="p">,</span> <span class="m">0</span><span class="p">},</span>
<span class="ss">dispatcher:</span> <span class="no">GenStage</span><span class="o">.</span><span class="no">BroadcastDispatcher</span><span class="p">}</span></code></pre></figure>
<h2 id="using-queuebroadcaster">Using QueueBroadcaster</h2>
<p>I’ll switch over to the implementaiton from <code class="highlighter-rouge">QueueBroadcaster</code> but in doing so I’ll need to remove this test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it should accept events"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">pid</span><span class="p">}</span> <span class="o">=</span> <span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">start_link</span><span class="p">()</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">notify</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="sd">"</span><span class="s2">Hi"</span><span class="p">)</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>Here <code class="highlighter-rouge">notify</code> waits for the event to be accepted by the subscribers. As this test sets up no subscribers it ends up timing out. This is no longer a suported behavior and it wasn’t a behavior that was important to me anyway. So I can remove the test. The behavior or accepting events and delivering them to subscribers is already tested in this test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">a subscribed observer is notified of all events"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">observer</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">TestObserver</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">updater</span><span class="p">}</span> <span class="o">=</span> <span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">start_link</span><span class="p">()</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">observer</span><span class="p">,</span> <span class="ss">to:</span> <span class="n">updater</span><span class="p">)</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">notify</span><span class="p">(</span><span class="n">updater</span><span class="p">,</span> <span class="sd">"</span><span class="s2">Hi"</span><span class="p">)</span>
<span class="n">assert</span> <span class="no">TestObserver</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">observer</span><span class="p">)</span> <span class="o">==</span> <span class="p">[</span><span class="sd">"</span><span class="s2">Hi"</span><span class="p">]</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">updater</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>My final version looks like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span>
<span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandler</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenStage</span>
<span class="k">def</span> <span class="n">start_link</span><span class="p">(),</span> <span class="k">do</span><span class="p">:</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">)</span>
<span class="k">def</span> <span class="n">stop</span><span class="p">(</span><span class="n">stage</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">stage</span><span class="p">)</span>
<span class="k">def</span> <span class="n">notify</span><span class="p">(</span><span class="n">handler</span><span class="p">,</span> <span class="n">event</span><span class="p">)</span> <span class="k">do</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">handler</span><span class="p">,</span> <span class="p">{</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">})</span>
<span class="k">end</span>
<span class="c1">## Callbacks</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="ss">:ok</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="p">{</span><span class="ss">:queue</span><span class="o">.</span><span class="n">new</span><span class="p">,</span> <span class="m">0</span><span class="p">},</span> <span class="ss">dispatcher:</span> <span class="no">GenStage</span><span class="o">.</span><span class="no">BroadcastDispatcher</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_call</span><span class="p">({</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">from</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">})</span> <span class="k">do</span>
<span class="n">dispatch_events</span><span class="p">(</span><span class="ss">:queue</span><span class="o">.</span><span class="ow">in</span><span class="p">({</span><span class="n">from</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">queue</span><span class="p">),</span> <span class="n">demand</span><span class="p">,</span> <span class="p">[])</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_demand</span><span class="p">(</span><span class="n">incoming_demand</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">})</span> <span class="k">do</span>
<span class="n">dispatch_events</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">incoming_demand</span> <span class="o">+</span> <span class="n">demand</span><span class="p">,</span> <span class="p">[])</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">dispatch_events</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">,</span> <span class="n">events</span><span class="p">)</span> <span class="k">do</span>
<span class="n">with</span> <span class="n">d</span> <span class="ow">when</span> <span class="n">d</span> <span class="o">></span> <span class="m">0</span> <span class="o"><-</span> <span class="n">demand</span><span class="p">,</span>
<span class="p">{</span><span class="n">item</span><span class="p">,</span> <span class="n">queue</span><span class="p">}</span> <span class="o">=</span> <span class="ss">:queue</span><span class="o">.</span><span class="n">out</span><span class="p">(</span><span class="n">queue</span><span class="p">),</span>
<span class="p">{</span><span class="ss">:value</span><span class="p">,</span> <span class="p">{</span><span class="n">from</span><span class="p">,</span> <span class="n">event</span><span class="p">}}</span> <span class="o"><-</span> <span class="n">item</span> <span class="k">do</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">reply</span><span class="p">(</span><span class="n">from</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">)</span>
<span class="n">dispatch_events</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span> <span class="o">-</span> <span class="m">1</span><span class="p">,</span> <span class="p">[</span><span class="n">event</span> <span class="o">|</span> <span class="n">events</span><span class="p">])</span>
<span class="k">else</span>
<span class="n">_</span> <span class="o">-></span> <span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="no">Enum</span><span class="o">.</span><span class="n">reverse</span><span class="p">(</span><span class="n">events</span><span class="p">),</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">}}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>I’ve maintained my <code class="highlighter-rouge">stop</code> and <code class="highlighter-rouge">notify</code> functions. Also, I haven’t named my server after the module. I expect to be able to have many instances of it.</p>
<p>Now, the last step is to actually call our <code class="highlighter-rouge">UpdateEventHandler</code> to make an update. We do this from the controller like this:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gu">@@ -2,6 +2,7 @@ defmodule PlayChannel.ToyController do
</span> use PlayChannel.Web, :controller
alias PlayChannel.Toy
<span class="gi">+ alias PlayChannel.Toy.UpdateEventHandler
</span>
plug :scrub_params, "toy" when action in [:create, :update]
<span class="gu">@@ -45,7 +46,9 @@ defmodule PlayChannel.ToyController do
</span>
case Repo.update(changeset) do
{:ok, toy} ->
<span class="gd">- PlayChannel.ToyChannel.broadcast_change(toy)
</span><span class="gi">+ UpdateEventHandler.notify(:toy_event_manager, {:update, toy})
</span>
conn
<span class="err"> |> put_flash(:info, "Toy updated successfully.")</span></code></pre></figure>
<p>Well, wait, that isn’t quite right. I’ve adapted this from the GenEvent version but in this case what’s <code class="highlighter-rouge">:toy_event_manager</code>?</p>
<p>I guess I do need to name my <code class="highlighter-rouge">GenStage</code>. Se we need to:</p>
<ol>
<li>Start an <code class="highlighter-rouge">UpdateEventHandler</code> in our supervision tree.</li>
<li>Add support for naming <code class="highlighter-rouge">UpdateEventHandler</code> instances.</li>
<li>Name our instance <code class="highlighter-rouge">:toy_event_manager</code></li>
<li>Register the channel with the <code class="highlighter-rouge">UpdateEventHandler</code></li>
</ol>
<p>Let’s start with a test for naming first.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">an updater can be referenced by name"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">observer</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">TestObserver</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="ss">name:</span> <span class="ss">:a_name</span><span class="p">)</span>
<span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">observer</span><span class="p">,</span> <span class="ss">to:</span> <span class="ss">:a_name</span><span class="p">)</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">notify</span><span class="p">(</span><span class="ss">:a_name</span><span class="p">,</span> <span class="sd">"</span><span class="s2">Hi"</span><span class="p">)</span>
<span class="n">assert</span> <span class="no">TestObserver</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">observer</span><span class="p">)</span> <span class="o">==</span> <span class="p">[</span><span class="sd">"</span><span class="s2">Hi"</span><span class="p">]</span>
<span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">updater</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>The implementation is easy enough. We’ll just allow any options accepted by <code class="highlighter-rouge">GenStage.startlink</code>:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gu">@@ -3,7 +3,7 @@ alias Experimental.GenStage
</span> defmodule PlayChannel.Toy.UpdateEventHandler do
use GenStage
<span class="gd">- def start_link(), do: GenStage.start_link(__MODULE__, :ok)
</span><span class="gi">+ def start_link(opts \\ []), do: GenStage.start_link(__MODULE__, :ok, opts)
</span> def stop(stage), do: GenStage.stop(stage)
<span class="err"> def notify(handler, event) do</span></code></pre></figure>
<p>and the tests pass. Now we just need to start an <code class="highlighter-rouge">UpdateEventHandler</code> in our supervision tree like this:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gu">@@ -1,5 +1,6 @@
</span>defmodule PlayChannel do
use Application
<span class="gi">+ alias PlayChannel.Toy.UpdateEventHandler
</span>
# See http://elixir-lang.org/docs/stable/elixir/Application.html
# for more information on OTP Applications
<span class="gu">@@ -13,6 +14,7 @@ defmodule PlayChannel do
</span>supervisor(PlayChannel.Repo, []),
# Here you could define other workers and supervisors as children
# worker(PlayChannel.Worker, [arg1, arg2, arg3]),
<span class="gi">+ worker(UpdateEventHandler, [[name: :toy_event_manager]])
</span>]
<span class="err"># See http://elixir-lang.org/docs/stable/elixir/Supervisor.html</span></code></pre></figure>
<h2 id="what-am-i-doing">What am I doing?</h2>
<p>Ugh, I think I’m totally lost here. In the GenEvent case, <code class="highlighter-rouge">UpdateEventHandler</code> was actually a separate GenServer, not part of the channel. And <code class="highlighter-rouge">GenEvent</code> played the role of the notification server. I need something like this again. I really should have reivewed the old post better. (And honestly, I think I made this same mistake with <code class="highlighter-rouge">GenEvent</code>)</p>
<p>Ok, we can fix this. I rename <code class="highlighter-rouge">UpdateEventHandler</code> to <code class="highlighter-rouge">NotificationCenter</code> which feels good. I like that name better.</p>
<p>Then, I extract what I’d written in the channel into a separate <code class="highlighter-rouge">GenStage</code>. And it is this new actor that should be named <code class="highlighter-rouge">UpdateEventHandler</code>. It looks like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span>
<span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandler</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenStage</span>
<span class="k">def</span> <span class="n">start_link</span><span class="p">(</span><span class="n">opts</span> <span class="p">\\</span> <span class="p">[]),</span> <span class="k">do</span><span class="p">:</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="k">def</span> <span class="n">stop</span><span class="p">(</span><span class="n">stage</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">stage</span><span class="p">)</span>
<span class="c1"># Callbacks</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:consumer</span><span class="p">,</span> <span class="no">nil</span><span class="p">,</span> <span class="ss">subscribe_to:</span> <span class="p">[</span><span class="ss">:toy_notifcation_center</span><span class="p">]}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_events</span><span class="p">(</span><span class="n">events</span><span class="p">,</span> <span class="n">_from</span><span class="p">,</span> <span class="n">_state</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">each</span> <span class="n">events</span><span class="p">,</span> <span class="k">fn</span>
<span class="p">{</span><span class="ss">:update</span><span class="p">,</span> <span class="n">toy</span><span class="p">}</span> <span class="o">-></span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">ToyChannel</span><span class="o">.</span><span class="n">broadcast_change</span><span class="p">(</span><span class="n">toy</span><span class="p">)</span>
<span class="k">end</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="p">[],</span> <span class="no">nil</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>There’s also one change that I’ve added here which I’ve taken from the <code class="highlighter-rouge">GenStage</code> announcment. The <code class="highlighter-rouge">init</code> function now uses <code class="highlighter-rouge">subscribe_to: [:toy_notifcation_center</code> to automatically subscribe to the notification center. <code class="highlighter-rouge">:toy_notifcation_center</code> is the name I setup for that notification center when setting up my supervisor, like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">worker</span><span class="p">(</span><span class="no">NotificationCenter</span><span class="p">,</span> <span class="p">[[</span><span class="ss">name:</span> <span class="ss">:toy_notifcation_center</span><span class="p">]]),</span>
<span class="n">worker</span><span class="p">(</span><span class="no">UpdateEventHandler</span><span class="p">,</span> <span class="p">[[</span><span class="ss">name:</span> <span class="ss">:toy_event_manager</span><span class="p">]])</span></code></pre></figure>
<p>The <code class="highlighter-rouge">subscribe_to</code> functionality in <code class="highlighter-rouge">GenStage</code> is nice in that I don’t have to write extra code after the supervisor is started to build up the connections.</p>
<p>And now it works!</p>
<h2 id="conclusion">Conclusion</h2>
<p>Well, that was quite a long post. Part of the length came from a few mistakes I made along the way but I know I learned something from them and I hope you can too.</p>
<p>It was good to explore using <code class="highlighter-rouge">GenStage</code> in an observer pattern and as a replacement for <code class="highlighter-rouge">GenEvent</code>. It felt like a lot of code to write though I guesss I went though several iterations with <code class="highlighter-rouge">QueueBroadcaster</code> / <code class="highlighter-rouge">NotificationCenter</code>. If I needed to do this again elsewhere the understanding I came to here would allow me to move a lot faster.</p>
<p>One thing I found really interesting is that some of the work involved in managing the queue in <code class="highlighter-rouge">NotificationCenter</code> reminded me of the <code class="highlighter-rouge">BlockingQueue</code> that I wrote a long time back in <a href="/blocking-queue/">Blocking Queue</a>. And the <code class="highlighter-rouge">BlockingQueue</code> is something I’ve used in many posts to solve similar problems to those solved by <code class="highlighter-rouge">GenStage</code>. Though I think in the <code class="highlighter-rouge">GenStage</code> model, the queue is primarily used up front at the producer stage and supply/demand between consumer stages is managed by the demand system build into <code class="highlighter-rouge">GenStage</code>.</p>
<p>Coming up in future posts, we still have a few items left on our <code class="highlighter-rouge">GenStage</code> TODO list:</p>
<ol>
<li>Experiment with <code class="highlighter-rouge">Flow</code></li>
<li>Rewrite Domain Scrapper to use <code class="highlighter-rouge">GenStage</code>.</li>
</ol>
<p>Next week I hope to learn more about <code class="highlighter-rouge">Flow</code>. I hope you’ll join me.</p>
<p><a href="http://learningelixir.joekain.com/using-gen-stage-to-notify-a-channel/">Read more...</a></p>
http://learningelixir.joekain.com/gen-stage2016-10-05T00:00:00-07:002016-10-05T00:00:00-07:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>Hello Learning Elixir readers! It’s has been a <strong>very</strong> long time since my last
post. I’ve been focused on other technologies and have taken
a new job. I want to thank my friend Doug Goldie for keeping me invovled with
Elixir during that time. And I want to thank my wife for encouraging me to get
back on track with this blog.</p>
<p>A lot has changed in Elixir since I last did any serious work with it. Elixir
1.3 was released, Elixir World 2016, and more. One of the things that caught my
eye is <a href="https://hexdocs.pm/gen_stage/Experimental.GenStage.html">GenStage</a>. Last
year, the plan was to build GenRouter and it looks like it’s now evolved into GenStage.</p>
<p>There’s a lot to learn with GenStage. To approach this subject, let’s start
with some questions:</p>
<ol>
<li>What is GenStage (and Flow for that matter)?</li>
<li>Where can I find out more information?</li>
<li>What would be a good project to try them out?</li>
</ol>
<p>For question #1 I started with a Google search and it turned up the official
<a href="http://elixir-lang.org/blog/2016/07/14/announcing-genstage/">GenStage Announcement</a>.
Wow, it was posted back in July, I have been gone a long time. Anyway, it gives
the following description:</p>
<blockquote>
<p>GenStage is a new Elixir behaviour for exchanging events with back-pressure between Elixir processes.</p>
</blockquote>
<p>But what does this mean? Let’s go over this definition piece by peice. First of
all, GenStage is for Elixir. That’s simple enough.</p>
<p>GenStage is a “behaviour” which is the OTP term for an interface. Specifically,
GenStage defines a set of functions or callbacks that must be implemented by a
process adopting the behaviour. GenStage may also provide defaults
implementations of those functions.</p>
<p>GenStage is “for exchanging events … between Elixir procsses”. Events? That’s
a change from what I remember GenRouter was intended to be. If I remember
correctly GenRouter was for streaming data through multiprocess pipelines. I
think this shift to events may have been a generalization from GenRouter.</p>
<p>GenStage does all of this with “back-pressure”. Back-pressure is a mechanism for
controlling the rate in a producer-consumer setup. If the producer and consumer
run assynchronously then it is likely that one can get ahead of the other. If
the consumer falls behind then back-pressure can be used to prevent the producer
from over-producing.</p>
<h3 id="background">Background</h3>
<p>The announcement gives this example</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">File</span><span class="o">.</span><span class="n">stream!</span><span class="p">(</span><span class="sd">"</span><span class="s2">path/to/some/file"</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">flat_map</span><span class="p">(</span><span class="k">fn</span> <span class="n">line</span> <span class="o">-></span>
<span class="no">String</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">,</span> <span class="sd">"</span><span class="s2"> "</span><span class="p">)</span>
<span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(%{},</span> <span class="k">fn</span> <span class="n">word</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-></span>
<span class="no">Map</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">word</span><span class="p">,</span> <span class="m">1</span><span class="p">,</span> <span class="o">&</span> <span class="nv">&1</span> <span class="o">+</span> <span class="m">1</span><span class="p">)</span>
<span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">()</span></code></pre></figure>
<p>to describe the motivation for GenStage. This is an example of a lazy data
transformation pipeline which is common in Elixir. But, this solution does not
leverage the concurrency afforded by the BEAM and modern CPUs.</p>
<p>The goal for GenStage is to enable concurrent processing of large datasets while
still retaining Elixir’s easy to understand style of data transformation pipelines.</p>
<h3 id="genstage">GenStage</h3>
<p>The announcement goes on to show and example of <code class="highlighter-rouge">GenStage</code> stages. First there
is a counter called <code class="highlighter-rouge">A</code></p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span>
<span class="k">defmodule</span> <span class="no">A</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenStage</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">counter</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="n">counter</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_demand</span><span class="p">(</span><span class="n">demand</span><span class="p">,</span> <span class="n">counter</span><span class="p">)</span> <span class="ow">when</span> <span class="n">demand</span> <span class="o">></span> <span class="m">0</span> <span class="k">do</span>
<span class="c1"># If the counter is 3 and we ask for 2 items, we will</span>
<span class="c1"># emit the items 3 and 4, and set the state to 5.</span>
<span class="n">events</span> <span class="o">=</span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">(</span><span class="n">counter</span><span class="o">..</span><span class="n">counter</span><span class="o">+</span><span class="n">demand</span><span class="o">-</span><span class="m">1</span><span class="p">)</span>
<span class="c1"># The events to emit is the second element of the tuple,</span>
<span class="c1"># the third being the state.</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">events</span><span class="p">,</span> <span class="n">counter</span> <span class="o">+</span> <span class="n">demand</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>which is a producer. One thing I was left wondering after reading the
announcement is the meaning of <code class="highlighter-rouge">:producer</code> in the <code class="highlighter-rouge">init/1</code> function. Is
<code class="highlighter-rouge">:producer</code> a special value recognized by GenStage or just the name for <code class="highlighter-rouge">A</code>?
Looking at the <a href="https://hexdocs.pm/gen_stage/Experimental.GenStage.html#c:init/1">docs</a>
I found:</p>
<blockquote>
<p>In case of successful start, this callback must return a tuple where the first element is the stage type, which is either a :producer, :consumer or :producer_consumer if it is taking both roles.</p>
</blockquote>
<p>So in fact <code class="highlighter-rouge">:producer</code> is a special value recognized by GenStage.</p>
<p>The rest of <code class="highlighter-rouge">A</code> is the <code class="highlighter-rouge">handle_demand/2</code> function. This is a callback which
GenStage will use to request more items from a producer type stage. The <code class="highlighter-rouge">demand</code>
argument is the requested, or demanded, number of events. The <code class="highlighter-rouge">counter</code> argument
is the current state for process <code class="highlighter-rouge">A</code>. Since stage <code class="highlighter-rouge">A</code> is a counter it
maintains the current count as its state. For each <code class="highlighter-rouge">handle_demand/2</code> call,
enough values to satisfy <code class="highlighter-rouge">demand</code> are returned and <code class="highlighter-rouge">counter</code> is incremented by
<code class="highlighter-rouge">demand</code>. In this way, <code class="highlighter-rouge">A</code> can return the subsequent set of values on the next call.</p>
<p>The announcment goes on to build a <code class="highlighter-rouge">:consumer</code> type stage which has a
<code class="highlighter-rouge">handle_events/3</code> callback. This function should process or store the passed
events and update the <code class="highlighter-rouge">GenStage</code>’s state.</p>
<p>There announcment also builds a <code class="highlighter-rouge">:producer_consumer</code> type stage. This type must
define both a <code class="highlighter-rouge">handle_demand/2</code> and <code class="highlighter-rouge">handle_events/3</code> callback.</p>
<h3 id="connecting-the-stages">Connecting the Stages</h3>
<p>The next step is to start the stages and then connect them using
<code class="highlighter-rouge">sync_subscribe/3</code>. This step seems a bit manual but it sounds like Flow will
privde an easier way to assembly these stages for simple cases.</p>
<p>The part I found most interesting was that multiple consumers can be connected
in order to create more concurrency. When I initially started reading the
announcment I was worried that GenStage would only allow for creating pipelined
concurrency which is not the most effective form of concurrency. And that’s
because a 5 stage pipeline only allows for 5 concurrent activities. But
GenStage seems to be much more flexible.</p>
<h3 id="genstage-use-cases">GenStage Use Cases</h3>
<p>The announcment describes a few use cases for GenStage.</p>
<h4 id="genstage-for-data-ingestion">GenStage for Data-Ingestion</h4>
<blockquote>
<p>One of the use cases for GenStage is to consume data from third-party systems.</p>
</blockquote>
<p>This sounds very similar to my <a href="/collecing-multiple-streams-in-elixir/">Domain Scraper</a> Domain Scraper experiment.
It would be an interesting exercise to go back and rewrite Domain Scraper using
GenStage.</p>
<h4 id="genstage-for-event-dispatching">GenStage for Event Dispatching</h4>
<p>Another use case:</p>
<blockquote>
<p>Another scenario where GenStage can be useful today is to replace cases where developers would have used GenEvent in the past.</p>
</blockquote>
<p>The announcment goes on to describe an advantage of GenStage over GenEvent:</p>
<blockquote>
<p>GenEvent, however, has one big flaw: the event manager and all event handlers run in the same process.</p>
</blockquote>
<p>That’s interesting. I hadn’t realized this limitation of GenEvent before.
GenStage seems like a big improvement in that case. When using GenEvent as a
notification system for the observer pattern there can be many observers. With
all handlers being run in the same process, all the handlers would have to be
run serially, sacrificing the concurrency that BEAM provides us.</p>
<p>Later on the announcment has a call to action for GenEvent users:</p>
<blockquote>
<p>First of all, now is the moment for the community to step in and try GenStage out. If you have used GenEvent in the past, can it be replaced by a GenStage? Similarly, if you were planning to implement an event handling system, give GenStage a try.</p>
</blockquote>
<p>Well I do have a project that uses GenEvent. Back in February I wrote about
<a href="http://learningelixir.joekain.com/using-genevent-to-notify-a-channel/">Using GenEvent to Notify a Channel of Updates in Elixir</a>.
This might be a good place for me to start experimenting with GenStage.</p>
<h3 id="a-look-at-flow">A look at Flow</h3>
<p>The end of the announcemnt gives a glimpse of Flow which allows the example
at the beginning of the announcment to be rewritten like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span><span class="o">.</span><span class="no">Flow</span>
<span class="c1"># Let's compile common patterns for performance</span>
<span class="n">empty_space</span> <span class="o">=</span> <span class="ss">:binary</span><span class="o">.</span><span class="n">compile_pattern</span><span class="p">(</span><span class="sd">"</span><span class="s2"> "</span><span class="p">)</span> <span class="c1"># NEW!</span>
<span class="no">File</span><span class="o">.</span><span class="n">stream!</span><span class="p">(</span><span class="sd">"</span><span class="s2">path/to/some/file"</span><span class="p">,</span> <span class="ss">read_ahead:</span> <span class="m">100_000</span><span class="p">)</span> <span class="c1"># NEW!</span>
<span class="o">|></span> <span class="no">Flow</span><span class="o">.</span><span class="n">from_enumerable</span><span class="p">()</span>
<span class="o">|></span> <span class="no">Flow</span><span class="o">.</span><span class="n">flat_map</span><span class="p">(</span><span class="k">fn</span> <span class="n">line</span> <span class="o">-></span>
<span class="n">for</span> <span class="n">word</span> <span class="o"><-</span> <span class="no">String</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">empty_space</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="n">word</span><span class="p">,</span> <span class="m">1</span><span class="p">}</span>
<span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Flow</span><span class="o">.</span><span class="n">partition_with</span><span class="p">(</span><span class="ss">storage:</span> <span class="ss">:ets</span><span class="p">)</span> <span class="c1"># NEW!</span>
<span class="o">|></span> <span class="no">Flow</span><span class="o">.</span><span class="n">reduce_by_key</span><span class="p">(</span><span class="o">&</span> <span class="nv">&1</span> <span class="o">+</span> <span class="nv">&2</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">()</span></code></pre></figure>
<p>This allows for asynchonrous processing in the data transformation pipeline.
Looking at the docs, it seems Flow is available and I’d like to try it out in
a future post.</p>
<h3 id="next-steps">Next Steps</h3>
<p>In this post we looked at the GenStage announcment and dug into certain parts
more deeply. We started out with a list of questions:</p>
<ol>
<li>What is GenStage (and Flow for that matter)?</li>
<li>Where can I find out more information?</li>
<li>What would be a good project to try them out?</li>
</ol>
<p>How did we do here?</p>
<ol>
<li>We answered this for GenStage. But, we’ve only touched on Flow.</li>
<li>We found out more information by reading the announcement and the docs. I
also plan to watch the
<a href="https://www.youtube.com/watch?v=apEWcpfsD2A">ElixirConf 2016 Keynote</a> which
I hear covers GenStage and Flow.</li>
<li>We identified two projects - GenEvent to notify a Phoenix Channel and Domain
Scrapper.</li>
</ol>
<p>In answering these questions we’ve left ourselves with some tasks to follow up
on. Here’s a list:</p>
<ol>
<li>Watch the ElixirConf 2016 Keynote</li>
<li>Use GenStage to notify a Phoenix Channel of new events.</li>
<li>Experiment with Flow</li>
<li>Rewrite Domain Scrapper to use GenStage.</li>
</ol>
<p>I hope to work through items 2-4 in future posts and I encourage you to watch the
Keynote, if you haven’t done so already, in the meantime.</p>
<p><a href="http://learningelixir.joekain.com/gen-stage/">Read more...</a></p>
http://learningelixir.joekain.com/canary-authorization-with-phoenix2016-04-26T00:00:00-07:002016-04-26T00:00:00-07:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>Last week I wrote a bit about how I use <a href="/using-guardian-and-canary-with-phoenix/">Guardian and Canary together</a>. This week I
want to go into more depth on how I’m using Canary.</p>
<h2 id="routes">Routes</h2>
<p>In the previous post, I went over a few of the routes I’m using. Let’s review the routes
for building and editing rentals:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">scope</span> <span class="sd">"</span><span class="s2">/"</span><span class="p">,</span> <span class="no">MyApp</span> <span class="k">do</span>
<span class="n">pipe_through</span> <span class="p">[</span><span class="ss">:browser</span><span class="p">,</span> <span class="ss">:browser_session</span><span class="p">,</span> <span class="ss">:require_login</span><span class="p">]</span>
<span class="n">resources</span> <span class="sd">"</span><span class="s2">/rentals"</span><span class="p">,</span> <span class="no">RentalController</span> <span class="k">do</span>
<span class="n">resources</span> <span class="sd">"</span><span class="s2">/addresses"</span><span class="p">,</span> <span class="no">AddressController</span><span class="p">,</span> <span class="ss">only:</span> <span class="p">[</span><span class="ss">:create</span><span class="p">,</span> <span class="ss">:update</span><span class="p">,</span> <span class="ss">:delete</span><span class="p">]</span>
<span class="n">resources</span> <span class="sd">"</span><span class="s2">/images"</span><span class="p">,</span> <span class="no">ImageController</span><span class="p">,</span> <span class="ss">only:</span> <span class="p">[</span><span class="ss">:create</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>We can create or edit rentals as well as create images and addresses for those rentals. That is,
<code class="highlighter-rouge">Rentals</code> have many <code class="highlighter-rouge">Images</code> and <code class="highlighter-rouge">Images</code> belong to a <code class="highlighter-rouge">Rental.</code> Similarly,
<code class="highlighter-rouge">Rentals</code> have one address and an <code class="highlighter-rouge">Address</code> belongs to a <code class="highlighter-rouge">Rental</code>.</p>
<h2 id="the-rental-controller">The Rental Controller</h2>
<p>Here’s part of the <code class="highlighter-rouge">Rental</code> controller that I’m using</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">AgoraBase</span><span class="o">.</span><span class="no">RentalController</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">AgoraBase</span><span class="o">.</span><span class="no">Web</span><span class="p">,</span> <span class="ss">:controller</span>
<span class="n">alias</span> <span class="no">AgoraBase</span><span class="o">.</span><span class="no">Rental</span>
<span class="n">plug</span> <span class="ss">:scrub_params</span><span class="p">,</span> <span class="sd">"</span><span class="s2">rental"</span> <span class="ow">when</span> <span class="n">action</span> <span class="ow">in</span> <span class="p">[</span><span class="ss">:create</span><span class="p">,</span> <span class="ss">:update</span><span class="p">]</span>
<span class="n">plug</span> <span class="ss">:authorize_resource</span><span class="p">,</span> <span class="ss">model:</span> <span class="no">Rental</span>
<span class="c1"># Actions follow ...</span>
<span class="k">end</span></code></pre></figure>
<p>I use Canary’s <code class="highlighter-rouge">:authorize_resource</code> plug to check if the user is authorized to
modify this rental. But how does Canary know which users are authorized and
which aren’t?</p>
<p>Canary uses the Canada package to do this and Canada defines a protocol that I
can implement for my <code class="highlighter-rouge">User</code> type. That protocol includes the function <code class="highlighter-rouge">can?/3</code>
which allows Canary to ask if my <code class="highlighter-rouge">User</code> <code class="highlighter-rouge">can?</code> perform a given action.</p>
<p>I’ve implemented the <code class="highlighter-rouge">Canada.Can</code> protocol for my <code class="highlighter-rouge">User</code> like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defimpl</span> <span class="no">Canada</span><span class="o">.</span><span class="no">Can</span><span class="p">,</span> <span class="ss">for:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">User</span> <span class="k">do</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">User</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Rental</span>
<span class="k">def</span> <span class="n">can?</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">action</span><span class="p">,</span> <span class="no">Rental</span><span class="p">)</span> <span class="ow">when</span> <span class="n">action</span> <span class="ow">in</span> <span class="p">[</span><span class="ss">:new</span><span class="p">,</span> <span class="ss">:create</span><span class="p">]</span> <span class="k">do</span>
<span class="n">user</span><span class="o">.</span><span class="n">role</span> <span class="o">==</span> <span class="sd">"</span><span class="s2">owner"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">can?</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">action</span><span class="p">,</span> <span class="n">rental</span> <span class="o">=</span> <span class="p">%</span><span class="no">Rental</span><span class="p">{})</span> <span class="k">do</span>
<span class="k">if</span> <span class="n">rental</span><span class="o">.</span><span class="n">owner_id</span> <span class="o">==</span> <span class="n">user</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="no">true</span><span class="p">,</span> <span class="k">else</span><span class="p">:</span> <span class="no">false</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">can?</span><span class="p">(</span><span class="n">subject</span><span class="p">,</span> <span class="n">action</span><span class="p">,</span> <span class="n">resource</span><span class="p">)</span> <span class="k">do</span>
<span class="k">raise</span> <span class="sd">"""
Unimplemented authorization check for User! To fix see below...
Please implement `can?` for User in #{__ENV__.file}.
The function should match:
subject: #{inspect subject}
action: #{action}
resource: #{inspect resource}
"""</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>I decided to store this file in web/modules/user_can.ex. And in this file I’ve implemented a
few different cases for <code class="highlighter-rouge">can?/3</code>. I also thought that it might be
confusing when things blow up due to cases being unimplemented. To make this
easier to debug I included the final case which raises
an exception with a helpful error message. The message will let me
know which file needs to be updated and what to match on. This has already been
pretty helpful for me as I’ve been implementing the app.</p>
<h2 id="the-image-controller">The Image Controller</h2>
<p>Here’s a portion of <code class="highlighter-rouge">ImageController</code> which is used to create new <code class="highlighter-rouge">Image</code> records:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">AgoraBase</span><span class="o">.</span><span class="no">ImageController</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">AgoraBase</span><span class="o">.</span><span class="no">Web</span><span class="p">,</span> <span class="ss">:controller</span>
<span class="n">alias</span> <span class="no">AgoraBase</span><span class="o">.</span><span class="no">Rental</span>
<span class="n">alias</span> <span class="no">AgoraBase</span><span class="o">.</span><span class="no">Rental</span><span class="o">.</span><span class="no">Image</span>
<span class="n">plug</span> <span class="ss">:scrub_params</span><span class="p">,</span> <span class="sd">"</span><span class="s2">image"</span>
<span class="n">plug</span> <span class="ss">:authorize_resource</span><span class="p">,</span> <span class="ss">model:</span> <span class="no">Rental</span><span class="p">,</span> <span class="ss">id_name:</span> <span class="sd">"</span><span class="s2">rental_id"</span><span class="p">,</span> <span class="ss">persisted:</span> <span class="no">true</span>
<span class="c1"># Actions follow ...</span>
<span class="k">end</span></code></pre></figure>
<p>In this case an <code class="highlighter-rouge">Image</code> belongs to a <code class="highlighter-rouge">Rental</code> so I’ve used Canary’s support for
nested models. I do this with the <code class="highlighter-rouge">:authorize_resource</code> plug by setting the
model explicitly to <code class="highlighter-rouge">Rental</code>, and specifing the id name as <code class="highlighter-rouge">"rental_id"</code>. This lets
Canary find the Rental in a route like /rentals/:rental_id/image. Finally, I
set the <code class="highlighter-rouge">:persisted</code> option to <code class="highlighter-rouge">true</code> to enable the nested resource support.</p>
<p>In this case Canary will invoke <code class="highlighter-rouge">can?/3</code> on the Rental itself. This is different
than checking if the <code class="highlighter-rouge">User</code> can create a <code class="highlighter-rouge">Rental</code>. In this case Canary will
pass in the <code class="highlighter-rouge">Rental</code> struct identified by <code class="highlighter-rouge">rental_id</code>. As we can see back in my
implementation of <code class="highlighter-rouge">can?/3</code></p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"> <span class="k">def</span> <span class="n">can?</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">action</span><span class="p">,</span> <span class="n">address</span> <span class="o">=</span> <span class="p">%</span><span class="no">Rental</span><span class="o">.</span><span class="no">Address</span><span class="p">{})</span> <span class="k">do</span>
<span class="n">address</span><span class="o">.</span><span class="n">rental</span><span class="o">.</span><span class="n">owner_id</span> <span class="o">==</span> <span class="n">user</span><span class="o">.</span><span class="n">id</span>
<span class="k">end</span></code></pre></figure>
<p>I check that the <code class="highlighter-rouge">User</code> is the owner of the rental.</p>
<h2 id="the-address-controller">The Address Controller</h2>
<p>A <code class="highlighter-rouge">Rental</code> also has an address. Here’s the controller I use to set it up:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">AddressController</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Web</span><span class="p">,</span> <span class="ss">:controller</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Rental</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Rental</span><span class="o">.</span><span class="no">Address</span>
<span class="n">plug</span> <span class="ss">:scrub_params</span><span class="p">,</span> <span class="sd">"</span><span class="s2">address"</span> <span class="ow">when</span> <span class="n">action</span> <span class="ow">in</span> <span class="p">[</span><span class="ss">:create</span><span class="p">,</span> <span class="ss">:update</span><span class="p">]</span>
<span class="n">plug</span> <span class="ss">:load_and_authorize_resource</span><span class="p">,</span> <span class="ss">model:</span> <span class="no">Rental</span><span class="p">,</span>
<span class="ss">id_name:</span> <span class="sd">"</span><span class="s2">rental_id"</span><span class="p">,</span>
<span class="ss">persisted:</span> <span class="no">true</span><span class="p">,</span>
<span class="ss">preload:</span> <span class="ss">:address</span>
<span class="k">def</span> <span class="n">create</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">address"</span> <span class="o">=></span> <span class="n">address_params</span><span class="p">})</span> <span class="k">do</span>
<span class="n">rental</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">assigns</span><span class="o">.</span><span class="n">rental</span>
<span class="c1"># ...</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">update</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">address"</span> <span class="o">=></span> <span class="n">address_params</span><span class="p">})</span> <span class="k">do</span>
<span class="n">rental</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">assigns</span><span class="o">.</span><span class="n">rental</span>
<span class="n">changeset</span> <span class="o">=</span> <span class="no">Address</span><span class="o">.</span><span class="n">changeset</span><span class="p">(</span><span class="n">rental</span><span class="o">.</span><span class="n">address</span><span class="p">,</span> <span class="n">address_params</span><span class="p">)</span>
<span class="c1"># ....</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Here I’ve done things a little differently. Again, I’ve use the nested resource
case. The permissions for creating or updating and <code class="highlighter-rouge">Address</code> are basically the
same as for creating an <code class="highlighter-rouge">Image</code>. But, I’ve also switched over to using the
<code class="highlighter-rouge">:load_and_authorize_resource</code> instead of just <code class="highlighter-rouge">:authorize_resource</code>. This
helps me to simplify my controller actions.</p>
<p>I like to think about <code class="highlighter-rouge">:load_and_authorize_resource</code> this way: Canary has to load
up the <code class="highlighter-rouge">Rental</code> data in order to check the authorization. My controller also
needs the <code class="highlighter-rouge">Rental</code> so I should be able to use the <code class="highlighter-rouge">Rental</code> copy loaded by Canary.
<code class="highlighter-rouge">:load_and_authorize_resource</code> lets me do exactly that. When Canary does the
authorization is stores the <code class="highlighter-rouge">Rental</code> in <code class="highlighter-rouge">conn.assigns.rental</code>. You can see where
I access this in the <code class="highlighter-rouge">create</code> action.</p>
<p>In <code class="highlighter-rouge">AddressController</code> I also need the <code class="highlighter-rouge">rental.address</code> association preloaded. The <code class="highlighter-rouge">:preload</code>
options to <code class="highlighter-rouge">:load_and_authorize_resource</code> takes care of this. And you can see
that I use <code class="highlighter-rouge">rental.address</code> when building a changelist in the <code class="highlighter-rouge">update</code> action.</p>
<p>I should have probably used <code class="highlighter-rouge">:load_and_authorize_resource</code> in my
<code class="highlighter-rouge">RentalController</code> as well. I may go back and do that later.</p>
<p><a href="http://learningelixir.joekain.com/canary-authorization-with-phoenix/">Read more...</a></p>
http://learningelixir.joekain.com/using-guardian-and-canary-with-phoenix2016-04-19T00:00:00-07:002016-04-19T00:00:00-07:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>As I’m mentioned before I have an ongoing project being developed using Elixir
and Phoenix. I’m using two third party packages that I wanted to write about in
this post. They are <a href="https://github.com/ueberauth/guardian">Guardian</a> and
<a href="https://github.com/cpjk/canary">Canary</a>.</p>
<h2 id="user-authentication-and-authorization">User Authentication and Authorization</h2>
<p>In my app I’m using Guardian for for authentication and session management.
Guardian has a lot of interesting functionality but at the moment I’m using it
for simple password based authentication of users. And, for authorization I’m
using the Canary library.</p>
<p>If you aren’t familiar with these two terms I’ll explain.</p>
<p>I use authentication to manage logins and sessions. The user authenicates him or
herself by providing a user name and password. Once authenticated a session is
created on the user’s behalf.</p>
<p>Once logged in the user has a certain set of privileges. For example, a user may
be allowed to modify certain resources that he or she owns. But that same user
may not be allowed to modify resources that belong to other users. Canary checks to
see if a user is authorized to cary out the actions that he or she requests.</p>
<h2 id="guardian-setup">Guardian Setup</h2>
<p>My Guardian configuration is pretty simple. I wrote up these pipelines for my
router for Guardian:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">pipeline</span> <span class="ss">:browser_session</span> <span class="k">do</span>
<span class="n">plug</span> <span class="no">Guardian</span><span class="o">.</span><span class="no">Plug</span><span class="o">.</span><span class="no">VerifySession</span>
<span class="n">plug</span> <span class="no">Guardian</span><span class="o">.</span><span class="no">Plug</span><span class="o">.</span><span class="no">LoadResource</span>
<span class="k">end</span>
<span class="n">pipeline</span> <span class="ss">:require_login</span> <span class="k">do</span>
<span class="n">plug</span> <span class="no">Guardian</span><span class="o">.</span><span class="no">Plug</span><span class="o">.</span><span class="no">EnsureAuthenticated</span><span class="p">,</span> <span class="ss">handler:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">GuardianErrorHandler</span>
<span class="k">end</span></code></pre></figure>
<p>Then I have created scopes that either support a logged in user or require a logged in
user. For example:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">scope</span> <span class="sd">"</span><span class="s2">/"</span><span class="p">,</span> <span class="no">MyApp</span> <span class="k">do</span>
<span class="n">pipe_through</span> <span class="p">[</span><span class="ss">:browser</span><span class="p">,</span> <span class="ss">:browser_session</span><span class="p">]</span>
<span class="n">get</span> <span class="sd">"</span><span class="s2">/"</span><span class="p">,</span> <span class="no">PageController</span><span class="p">,</span> <span class="ss">:index</span>
<span class="n">resources</span> <span class="sd">"</span><span class="s2">/search"</span><span class="p">,</span> <span class="no">SearchController</span><span class="p">,</span> <span class="ss">only:</span> <span class="p">[</span><span class="ss">:index</span><span class="p">,</span> <span class="ss">:show</span><span class="p">]</span>
<span class="k">end</span></code></pre></figure>
<p>This scope user the <code class="highlighter-rouge">:browser_session</code> pipeine to load a session if one exists.
But it doesn’t require a session. This way both logged in users or guest users
can browse the home page and search.</p>
<p>But the resouces in this scope require a user to be logged in:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">scope</span> <span class="sd">"</span><span class="s2">/"</span><span class="p">,</span> <span class="no">MyApp</span> <span class="k">do</span>
<span class="n">pipe_through</span> <span class="p">[</span><span class="ss">:browser</span><span class="p">,</span> <span class="ss">:browser_session</span><span class="p">,</span> <span class="ss">:require_login</span><span class="p">]</span>
<span class="n">resources</span> <span class="sd">"</span><span class="s2">/rentals"</span><span class="p">,</span> <span class="no">RentalController</span> <span class="k">do</span>
<span class="n">resources</span> <span class="sd">"</span><span class="s2">/images"</span><span class="p">,</span> <span class="no">ImageController</span><span class="p">,</span> <span class="ss">only:</span> <span class="p">[</span><span class="ss">:create</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Here, I add the <code class="highlighter-rouge">:require_login</code> pipeline which uses Guardian’s <code class="highlighter-rouge">EnsureAuthenticated</code>
plug to require a session. Errors are passed off to my
<code class="highlighter-rouge">MyApp.GuardianErrorHandler</code> module which is responsible for redirecting guest
users to <code class="highlighter-rouge">/login</code>. This is a simple module that looks like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">GuardianErrorHandler</span> <span class="k">do</span>
<span class="kn">import</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Router</span><span class="o">.</span><span class="no">Helpers</span>
<span class="k">def</span> <span class="n">unauthenticated</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">_params</span><span class="p">)</span> <span class="k">do</span>
<span class="n">conn</span>
<span class="o">|></span> <span class="no">Phoenix</span><span class="o">.</span><span class="no">Controller</span><span class="o">.</span><span class="n">put_flash</span><span class="p">(</span><span class="ss">:error</span><span class="p">,</span>
<span class="sd">"</span><span class="s2">You must be logged in to access that page."</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Phoenix</span><span class="o">.</span><span class="no">Controller</span><span class="o">.</span><span class="n">redirect</span><span class="p">(</span><span class="ss">to:</span> <span class="n">login_path</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="ss">:new</span><span class="p">))</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<h2 id="my-canary-setup">My Canary Setup</h2>
<p>I use canary in my controllers to check if a user has authorization to view or
modify a given resource.</p>
<p>For example, above we saw routes that edit rentals and create images for those rentals. And that these routes require a logged in user. Well, we need more than just a logged in user. The user needs to own those resources.</p>
<p>Here’s an example of how I use Canary to authorize for Images:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">RentalController</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Web</span><span class="p">,</span> <span class="ss">:controller</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Rental</span>
<span class="n">plug</span> <span class="ss">:scrub_params</span><span class="p">,</span> <span class="sd">"</span><span class="s2">rental"</span> <span class="ow">when</span> <span class="n">action</span> <span class="ow">in</span> <span class="p">[</span><span class="ss">:create</span><span class="p">,</span> <span class="ss">:update</span><span class="p">]</span>
<span class="n">plug</span> <span class="ss">:authorize_resource</span><span class="p">,</span> <span class="ss">model:</span> <span class="no">Rental</span>
<span class="c1"># Actions follow ...</span>
<span class="k">end</span></code></pre></figure>
<p>Canary’s <code class="highlighter-rouge">:authorize_resource</code> plug checks if the current user is allowed to act on the a <code class="highlighter-rouge">Rental</code>. This of course depends on a <code class="highlighter-rouge">can?</code> function. Here’s part of the <code class="highlighter-rouge">can?</code> function I’ve written:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">can?</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">action</span><span class="p">,</span> <span class="no">Rental</span><span class="p">)</span> <span class="ow">when</span> <span class="n">action</span> <span class="ow">in</span> <span class="p">[</span><span class="ss">:new</span><span class="p">,</span> <span class="ss">:create</span><span class="p">]</span> <span class="k">do</span>
<span class="n">user</span><span class="o">.</span><span class="n">role</span> <span class="o">==</span> <span class="sd">"</span><span class="s2">owner"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">can?</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">action</span><span class="p">,</span> <span class="n">rental</span> <span class="o">=</span> <span class="p">%</span><span class="no">Rental</span><span class="p">{})</span> <span class="k">do</span>
<span class="n">rental</span><span class="o">.</span><span class="n">owner_id</span> <span class="o">==</span> <span class="n">user</span><span class="o">.</span><span class="n">id</span>
<span class="k">end</span></code></pre></figure>
<p>Here I decide if the user is authorized depending on the action.</p>
<h2 id="using-canary-with-guardian">Using Canary with Guardian</h2>
<p>All of the above seems pretty great, but it doesn’t work!</p>
<p>One of the problems I had making Guardian and Canary play well together was that
Canary depends on being able to find the current user in the connection.
Guardian maintains the current user and it can be accessed by calling
<code class="highlighter-rouge">Guardian.Plug.current_resource(conn)</code>. But, Canary expects the current user in
the <code class="highlighter-rouge">:current_user</code> assign in the conn.</p>
<p>To make this work I wrote up another small plug:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Plug</span><span class="o">.</span><span class="no">CurrentUser</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">opts</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">opts</span>
<span class="k">def</span> <span class="n">call</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">_opts</span><span class="p">)</span> <span class="k">do</span>
<span class="n">current_user</span> <span class="o">=</span> <span class="no">Guardian</span><span class="o">.</span><span class="no">Plug</span><span class="o">.</span><span class="n">current_resource</span><span class="p">(</span><span class="n">conn</span><span class="p">)</span>
<span class="no">Plug</span><span class="o">.</span><span class="no">Conn</span><span class="o">.</span><span class="n">assign</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="ss">:current_user</span><span class="p">,</span> <span class="n">current_user</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>This plug simply queries the current user from Guardian and put’s it into the <code class="highlighter-rouge">:current_user</code> assign in the <code class="highlighter-rouge">conn</code>. This copies the data from Guardian’s location to the location Canary expects.</p>
<p>I add this plug to my <code class="highlighter-rouge">:require_login</code> pipeline</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">pipeline</span> <span class="ss">:require_login</span> <span class="k">do</span>
<span class="n">plug</span> <span class="no">Guardian</span><span class="o">.</span><span class="no">Plug</span><span class="o">.</span><span class="no">EnsureAuthenticated</span><span class="p">,</span> <span class="ss">handler:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">GuardianErrorHandler</span>
<span class="n">plug</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Plug</span><span class="o">.</span><span class="no">CurrentUser</span>
<span class="k">end</span></code></pre></figure>
<p>With this plug Canary can access <code class="highlighter-rouge">:current_user</code> as an assign and all is well. I
also use the <code class="highlighter-rouge">:current_user</code> assign in my own code. My controllers don’t need to
know about Guardian.</p>
<h2 id="next-steps">Next Steps</h2>
<p>This post focused on how I use Guardian and Canary together to authenticate and
authorize in my app. I covered all of my Guardian usage and what I had to do in
order to use the two libraries together. But, there’s a bit more to my Canary
usage that I hope to write about next week.</p>
<p><a href="http://learningelixir.joekain.com/using-guardian-and-canary-with-phoenix/">Read more...</a></p>
http://learningelixir.joekain.com/elixir-arc-with-a-single-module2016-04-12T00:00:00-07:002016-04-12T00:00:00-07:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>Dear readers, I haven’t posted on Learning Elixir in almost a month. Please know
that I’m working to get back onto a weekly posting schedule. I have an ongoing
project using Elixir and Phoenix which is keeping mey busy, but it should also
provide a number of interesting topics for this blog. Speaking of which…</p>
<p>In this project, I needed to upload images and store them on Amazon S3. The
<a href="https://github.com/stavro/arc">Arc library</a> has great support for doing this.</p>
<p>When setting up my image support I had written an <code class="highlighter-rouge">Image</code> module that contained
the Ecto Schema for my images. It also contained a few useful functions for
working with those images.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Image</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Web</span><span class="p">,</span> <span class="ss">:model</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Uploaders</span>
<span class="no">Schema</span> <span class="sd">"</span><span class="s2">images"</span> <span class="k">do</span>
<span class="n">field</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">field</span> <span class="ss">:upload</span><span class="p">,</span> <span class="ss">:any</span><span class="p">,</span> <span class="ss">virtual:</span> <span class="no">true</span>
<span class="n">belongs_to</span> <span class="ss">:rental</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Rental</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="nv">@required_fields</span> <span class="err">~</span><span class="n">w</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="nv">@optional_fields</span> <span class="err">~</span><span class="n">w</span><span class="p">()</span>
<span class="k">def</span> <span class="n">changeset</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">params</span> <span class="p">\\</span> <span class="p">%{})</span> <span class="k">do</span>
<span class="n">model</span>
<span class="o">|></span> <span class="n">cast</span><span class="p">(</span><span class="n">params</span><span class="p">,</span> <span class="err">~</span><span class="n">w</span><span class="p">(</span><span class="n">upload</span><span class="p">),</span> <span class="p">[])</span>
<span class="o">|></span> <span class="n">put_name</span>
<span class="o">|></span> <span class="n">cast</span><span class="p">(</span><span class="n">params</span><span class="p">,</span> <span class="nv">@required_fields</span><span class="p">,</span> <span class="nv">@optional_fields</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">put_name</span><span class="p">(</span><span class="n">changeset</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="n">changeset</span> <span class="k">do</span>
<span class="p">%</span><span class="no">Ecto</span><span class="o">.</span><span class="no">Changeset</span><span class="p">{</span>
<span class="ss">valid?:</span> <span class="no">true</span><span class="p">,</span>
<span class="ss">changes:</span> <span class="p">%{</span><span class="ss">upload:</span> <span class="p">%</span><span class="no">Plug</span><span class="o">.</span><span class="no">Upload</span><span class="p">{</span><span class="ss">content_type:</span> <span class="sd">"</span><span class="s2">image/"</span> <span class="o"><></span> <span class="n">_</span><span class="p">,</span> <span class="ss">filename:</span> <span class="n">name</span><span class="p">}}</span>
<span class="p">}</span> <span class="o">-></span>
<span class="n">put_change</span><span class="p">(</span><span class="n">changeset</span><span class="p">,</span> <span class="ss">:name</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="n">_</span> <span class="o">-></span>
<span class="n">changeset</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">store</span><span class="p">(%</span><span class="no">Plug</span><span class="o">.</span><span class="no">Upload</span><span class="p">{}</span> <span class="o">=</span> <span class="n">upload</span><span class="p">,</span> <span class="n">image</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Uploaders</span><span class="o">.</span><span class="no">RentalImage</span><span class="o">.</span><span class="n">store</span><span class="p">({</span><span class="n">upload</span><span class="p">,</span> <span class="n">image</span><span class="p">})</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">url</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">version</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Uploaders</span><span class="o">.</span><span class="no">RentalImage</span><span class="o">.</span><span class="n">url</span><span class="p">({</span><span class="n">image</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">image</span><span class="p">},</span> <span class="n">version</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>This <code class="highlighter-rouge">Image</code> module has all the basic model stuff like the schema and
<code class="highlighter-rouge">changeset/2</code> function. It also includes functions <code class="highlighter-rouge">store/2</code> and <code class="highlighter-rouge">url/2</code> which
can be used to store a new image to S3 and to retreive the url for a previously
stored image, respectively. Both functions rely on Arc functionality through
<code class="highlighter-rouge">Uploaders.RentalImage</code>. They also hide the Arc from users of the <code class="highlighter-rouge">Image</code>
module. Those users only need to interact with the <code class="highlighter-rouge">Image</code> module and can ignore
the <code class="highlighter-rouge">Uploaders.RentalImage</code> module altogether.</p>
<p>The <code class="highlighter-rouge">Uploaders.RentalImage</code> module follows the the Arc documentation, and
configures the way we want to use Arc.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Uploaders</span><span class="o">.</span><span class="no">RentalImage</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Arc</span><span class="o">.</span><span class="no">Definition</span>
<span class="nv">@acl</span> <span class="ss">:public_read</span>
<span class="nv">@versions</span> <span class="p">[</span><span class="ss">:original</span><span class="p">,</span> <span class="ss">:show</span><span class="p">,</span> <span class="ss">:thumb</span><span class="p">]</span>
<span class="nv">@heights</span> <span class="p">%{</span>
<span class="ss">show:</span> <span class="m">315</span><span class="p">,</span>
<span class="ss">thumb:</span> <span class="m">30</span>
<span class="p">}</span>
<span class="k">def</span> <span class="n">validate</span><span class="p">({</span><span class="n">file</span><span class="p">,</span> <span class="n">_</span><span class="p">})</span> <span class="k">do</span>
<span class="err">~</span><span class="n">w</span><span class="p">(</span><span class="o">.</span><span class="n">jpg</span> <span class="o">.</span><span class="n">jpeg</span> <span class="o">.</span><span class="n">gif</span> <span class="o">.</span><span class="n">png</span><span class="p">)</span> <span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">member?</span><span class="p">(</span><span class="no">Path</span><span class="o">.</span><span class="n">extname</span><span class="p">(</span><span class="n">file</span><span class="o">.</span><span class="n">file_name</span><span class="p">))</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">transform</span><span class="p">(</span><span class="ss">:thumb</span><span class="p">,</span> <span class="n">_file</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:convert</span><span class="p">,</span> <span class="sd">"</span><span class="s2">-thumbnail x</span><span class="si">#{</span><span class="nv">@heights</span><span class="p">[</span><span class="ss">:thumb</span><span class="p">]</span><span class="si">}</span><span class="s2"> -gravity center -format jpg"</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">transform</span><span class="p">(</span><span class="ss">:show</span><span class="p">,</span> <span class="n">_file</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:convert</span><span class="p">,</span> <span class="sd">"</span><span class="s2">-strip -resize x</span><span class="si">#{</span><span class="nv">@heights</span><span class="p">[</span><span class="ss">:show</span><span class="p">]</span><span class="si">}</span><span class="s2"> -gravity center -format png"</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">storage_dir</span><span class="p">(</span><span class="n">version</span><span class="p">,</span> <span class="p">{</span><span class="n">_</span><span class="p">,</span> <span class="n">image</span><span class="p">})</span> <span class="k">do</span>
<span class="sd">"</span><span class="s2">uploads/rentals/</span><span class="si">#{</span><span class="n">image</span><span class="o">.</span><span class="n">rental_id</span><span class="si">}</span><span class="s2">/images/</span><span class="si">#{</span><span class="n">image</span><span class="o">.</span><span class="n">id</span><span class="si">}</span><span class="s2">/</span><span class="si">#{</span><span class="n">version</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">filename</span><span class="p">(</span><span class="n">_version</span><span class="p">,</span> <span class="p">{</span><span class="n">file</span><span class="p">,</span> <span class="n">_</span><span class="p">})</span> <span class="k">do</span>
<span class="no">Path</span><span class="o">.</span><span class="n">rootname</span><span class="p">(</span><span class="n">file</span><span class="o">.</span><span class="n">file_name</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>This module provides a number of functions that Arc expects to be able to call
to determine how we want Arc to behave. For example, <code class="highlighter-rouge">storage_dir/2</code> is called
to determine the directory in which to store a given Image.</p>
<p>I’m also following Arc’s use case of having the S3 resource attached to my <code class="highlighter-rouge">Image</code>
model. Hence the tuple format for the parameters to <code class="highlighter-rouge">validate/1</code> and <code class="highlighter-rouge">filename/2</code>.</p>
<h2 id="confusing">Confusing?</h2>
<p>When I showed this to my coworker he was a little confused about the role of
<code class="highlighter-rouge">Uploaders.RentalImage</code>. While he might be going against Arc’s perscription, I
think he had a valid point: why does the uploader need to be a separate module?</p>
<p>When we started on a large refactoring of our application I decided to see
what it would be like to combine <code class="highlighter-rouge">Image</code> and <code class="highlighter-rouge">Uploaders.RentalImage</code> in a single
module.</p>
<p>My plan was to integrate all the code into a single module and then see what
could be simplified or improved.</p>
<h2 id="a-single-module-with-model-and-arc-uploader">A Single Module with Model and Arc Uploader</h2>
<p>I did as a planned and put everything together. I ended up with this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Rental</span><span class="o">.</span><span class="no">Image</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Web</span><span class="p">,</span> <span class="ss">:model</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Rental</span>
<span class="n">schema</span> <span class="sd">"</span><span class="s2">images"</span> <span class="k">do</span>
<span class="n">field</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">field</span> <span class="ss">:upload</span><span class="p">,</span> <span class="ss">:any</span><span class="p">,</span> <span class="ss">virtual:</span> <span class="no">true</span>
<span class="n">belongs_to</span> <span class="ss">:rental</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Rental</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="nv">@required_fields</span> <span class="err">~</span><span class="n">w</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="nv">@optional_fields</span> <span class="err">~</span><span class="n">w</span><span class="p">()</span>
<span class="k">def</span> <span class="n">changeset</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">params</span> <span class="p">\\</span> <span class="p">%{})</span> <span class="k">do</span>
<span class="n">model</span>
<span class="o">|></span> <span class="n">cast</span><span class="p">(</span><span class="n">params</span><span class="p">,</span> <span class="err">~</span><span class="n">w</span><span class="p">(</span><span class="n">upload</span><span class="p">),</span> <span class="p">[])</span>
<span class="o">|></span> <span class="n">put_name</span>
<span class="o">|></span> <span class="n">cast</span><span class="p">(</span><span class="n">params</span><span class="p">,</span> <span class="nv">@required_fields</span><span class="p">,</span> <span class="nv">@optional_fields</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">put_name</span><span class="p">(</span><span class="n">changeset</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="n">changeset</span> <span class="k">do</span>
<span class="p">%</span><span class="no">Ecto</span><span class="o">.</span><span class="no">Changeset</span><span class="p">{</span>
<span class="ss">valid?:</span> <span class="no">true</span><span class="p">,</span>
<span class="ss">changes:</span> <span class="p">%{</span>
<span class="ss">upload:</span> <span class="p">%</span><span class="no">Plug</span><span class="o">.</span><span class="no">Upload</span><span class="p">{</span><span class="ss">content_type:</span> <span class="sd">"</span><span class="s2">image/"</span> <span class="o"><></span> <span class="n">_</span><span class="p">,</span> <span class="ss">filename:</span> <span class="n">name</span><span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span> <span class="o">-></span>
<span class="n">put_change</span><span class="p">(</span><span class="n">changeset</span><span class="p">,</span> <span class="ss">:name</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="n">_</span> <span class="o">-></span>
<span class="n">changeset</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">store</span><span class="p">(%</span><span class="no">Plug</span><span class="o">.</span><span class="no">Upload</span><span class="p">{}</span> <span class="o">=</span> <span class="n">upload</span><span class="p">,</span> <span class="n">image</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">store</span><span class="p">({</span><span class="n">upload</span><span class="p">,</span> <span class="n">image</span><span class="p">})</span>
<span class="k">def</span> <span class="n">url</span><span class="p">(%</span><span class="no">Rental</span><span class="o">.</span><span class="no">Image</span><span class="p">{}</span> <span class="o">=</span> <span class="n">image</span><span class="p">,</span> <span class="n">version</span><span class="p">)</span> <span class="k">do</span>
<span class="n">url</span><span class="p">({</span><span class="n">image</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">image</span><span class="p">},</span> <span class="n">version</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1">########################################################################</span>
<span class="c1"># The rest of this file is the Arc definition for uploading</span>
<span class="c1">########################################################################</span>
<span class="kn">use</span> <span class="no">Arc</span><span class="o">.</span><span class="no">Definition</span>
<span class="nv">@acl</span> <span class="ss">:public_read</span>
<span class="nv">@versions</span> <span class="p">[</span><span class="ss">:original</span><span class="p">,</span> <span class="ss">:slider</span><span class="p">,</span> <span class="ss">:card</span><span class="p">]</span>
<span class="nv">@heights</span> <span class="p">%{</span>
<span class="ss">card:</span> <span class="m">230</span><span class="p">,</span>
<span class="ss">slider:</span> <span class="m">358</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">def</span> <span class="n">validate</span><span class="p">({</span><span class="n">file</span><span class="p">,</span> <span class="n">_</span><span class="p">})</span> <span class="k">do</span>
<span class="err">~</span><span class="n">w</span><span class="p">(</span><span class="o">.</span><span class="n">jpg</span> <span class="o">.</span><span class="n">jpeg</span> <span class="o">.</span><span class="n">gif</span> <span class="o">.</span><span class="n">png</span><span class="p">)</span> <span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">member?</span><span class="p">(</span><span class="no">Path</span><span class="o">.</span><span class="n">extname</span><span class="p">(</span><span class="n">file</span><span class="o">.</span><span class="n">file_name</span><span class="p">))</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">transform</span><span class="p">(</span><span class="ss">:original</span><span class="p">,</span> <span class="n">_file</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="ss">:noaction</span>
<span class="k">def</span> <span class="n">transform</span><span class="p">(</span><span class="ss">:slider</span><span class="p">,</span> <span class="n">_file</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:convert</span><span class="p">,</span> <span class="sd">"</span><span class="s2">-strip -resize 612x358 -extent 612x358 -background black -gravity center -format jpg"</span><span class="p">,</span> <span class="ss">:jpg</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">transform</span><span class="p">(</span><span class="n">tag</span><span class="p">,</span> <span class="n">_file</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:convert</span><span class="p">,</span> <span class="sd">"</span><span class="s2">-strip -resize x</span><span class="si">#{</span><span class="nv">@heights</span><span class="p">[</span><span class="n">tag</span><span class="p">]</span><span class="si">}</span><span class="s2"> -gravity center -format jpg"</span><span class="p">,</span> <span class="ss">:jpg</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">storage_dir</span><span class="p">(</span><span class="n">version</span><span class="p">,</span> <span class="p">{</span><span class="n">_</span><span class="p">,</span> <span class="n">image</span><span class="p">})</span> <span class="k">do</span>
<span class="sd">"</span><span class="s2">uploads/rentals/</span><span class="si">#{</span><span class="n">image</span><span class="o">.</span><span class="n">rental_id</span><span class="si">}</span><span class="s2">/images/</span><span class="si">#{</span><span class="n">image</span><span class="o">.</span><span class="n">id</span><span class="si">}</span><span class="s2">/</span><span class="si">#{</span><span class="n">version</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">filename</span><span class="p">(</span><span class="n">_version</span><span class="p">,</span> <span class="p">{</span><span class="n">file</span><span class="p">,</span> <span class="n">_</span><span class="p">}),</span> <span class="k">do</span><span class="p">:</span> <span class="no">Path</span><span class="o">.</span><span class="n">rootname</span><span class="p">(</span><span class="n">file</span><span class="o">.</span><span class="n">file_name</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>Note, that there are a few differences beyond just combining the two files. For
example, I am now supporting a different set of image versions and transforms (<code class="highlighter-rouge">:card</code> and
<code class="highlighter-rouge">:slider</code> instead of <code class="highlighter-rouge">thumb</code> and <code class="highlighter-rouge">:show</code>).</p>
<p>I drew a small separation between the parts using a comment. Should I do more
here?</p>
<p>There wasn’t much in terms of simplification to do with this combination aside
from being able to drop <code class="highlighter-rouge">Uploaders.RentalImage</code> from calls to <code class="highlighter-rouge">store/1</code> and
<code class="highlighter-rouge">url/2</code>.</p>
<p>I still maintained the custom <code class="highlighter-rouge">store/2</code> and <code class="highlighter-rouge">url/2</code> functions, wrappers around
the Arc functions. I find their interfaces simpler than Arc’s tuple format.</p>
<p>Here’s an example of how I use the <code class="highlighter-rouge">url/2</code> function from the view to put the
<code class="highlighter-rouge">:slider</code> version of an image into an image slider:</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><img</span> <span class="na">src=</span><span class="s">"<%= Image.url(image, :slider) %>"</span>
<span class="na">alt=</span><span class="s">""</span> <span class="na">data-bgposition=</span><span class="s">"center bottom"</span> <span class="na">data-bgfit=</span><span class="s">"cover"</span>
<span class="na">data-bgrepeat=</span><span class="s">"no-repeat"</span> <span class="na">data-bgparallax=</span><span class="s">"10"</span>
<span class="na">class=</span><span class="s">"rev-slidebg"</span> <span class="na">data-no-retina</span><span class="nt">></span></code></pre></figure>
<p>I like just being able to pass <code class="highlighter-rouge">image</code> instead of having to build a tuple like
<code class="highlighter-rouge"><span class="p">{</span><span class="err">image.name,</span><span class="w"> </span><span class="err">image</span><span class="p">}</span></code>. The tuple has redundnacy so I prefere to write out the
tuple once within my <code class="highlighter-rouge">url/2</code> function and avoid it in all other uses.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Overall I’m pretty happy with this. On one hand using a separate uploader module
separates distinct concerns. But on the other hand the code involved isn’t that
large and keeping it all in one module helps keep all my Image related code
together in one place.</p>
<p>The Arc uploader concerns itself mostly with naming files and given that the
model needs to be aware of some of these naming conventions in order to store
the name in the database I think it makes sense to keep these things together.</p>
<p>If this functionality grows over time then splitting this into multiple modules
might make sense. But I’m pretty happy with what I have right now.</p>
<p><a href="http://learningelixir.joekain.com/elixir-arc-with-a-single-module/">Read more...</a></p>
http://learningelixir.joekain.com/none-of-the-associations-in-ecto2016-03-15T00:00:00-07:002016-03-15T00:00:00-07:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>I’ve been working with Ecto a lot lately and wanted to spend this post writing up something I’ve learned while trying to put together a query. I’m by no means a database expert and this is one of the more complex queries I’ve had to write so far.</p>
<p>In the private project I’m working on, we have a database schema that consists of <code class="highlighter-rouge">rentals</code> that are marked unavailable on certain dates. These unavailable date ranges are stored in an <code class="highlighter-rouge">Unavailability</code> model and associated through a <code class="highlighter-rouge">has_many</code> / <code class="highlighter-rouge">belongs_to</code> relationship. That is, a <code class="highlighter-rouge">Rental</code> has many <code class="highlighter-rouge">unavalable_date_ranges</code> and <code class="highlighter-rouge">Unavailability</code> belongs to a <code class="highlighter-rouge">Rental</code> like this:</p>
<!-- _includes/image.html -->
<!-- from http://codingtips.kanishkkunal.in/image-caption-jekyll/ -->
<div class="image-wrapper" align="center">
<img src="/images/rental-unavailability.png" alt="Rental and Unavailability" />
</div>
<p>When trying to find a rental I want the user to be able to search (query) for <code class="highlighter-rouge">rentals</code> that are available on a given day or range of days. This means finding <code class="highlighter-rouge">rentals</code> that have no associated <code class="highlighter-rouge">Unavailability</code> records that overlap the desired days.</p>
<p>A while back I had written a query that turned out not to work. It only worked for very simple cases but failed when a <code class="highlighter-rouge">rental</code> had more than one associated <code class="highlighter-rouge">Unavailability</code>. I set out to fix this.</p>
<h2 id="research-and-experimentation">Research and Experimentation</h2>
<p>I’m not an SQL or Ecto expert so I set out to research what I needed to learn in order to solve my problem. After some googling around I came across the blog post <a href="https://www.pagerduty.com/blog/sql-left-join-trick/">“All, None, and One: The SQL Left Join Trick”</a>. This post taught me, as it promissed, how to use a left join to find rows with no matching associations. I highly recommend reading the post but I’ll summarize what I took away from it.</p>
<p>A left join between <code class="highlighter-rouge">Rental</code> and <code class="highlighter-rouge">Unavailability</code> will contain all of the rows from <code class="highlighter-rouge">Rental</code> even if they have no matching <code class="highlighter-rouge">unavailable_date_ranges</code>. And we can add additional criteria so that the results has only those rentals with no matching <code class="highlighter-rouge">unavalable_date_ranges</code>, that is those that are available.</p>
<p>I played around with this technique and was able to build up this query:</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">rentals</span>
<span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="n">unavailabilities</span>
<span class="k">ON</span> <span class="p">(</span><span class="n">unavailabilities</span><span class="p">.</span><span class="n">rental_id</span> <span class="o">=</span> <span class="n">rentals</span><span class="p">.</span><span class="n">id</span><span class="p">)</span>
<span class="k">WHERE</span> <span class="n">rental_id</span> <span class="k">IS</span> <span class="k">NULL</span><span class="p">;</span></code></pre></figure>
<p>This would return all the colums for <code class="highlighter-rouge">rentals</code> where there are no associated <code class="highlighter-rouge">unavalable_date_ranges</code>. The <code class="highlighter-rouge">ON</code> clause joins by the association. The <code class="highlighter-rouge">WHERE</code> clause filters the result to those rentals without any matching <code class="highlighter-rouge">unavalable_date_ranges</code>.</p>
<p>But this doesn’t get me exactly what I want. I need to add in a condition to check for overlap. I know how to write the condition:</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="p">(</span><span class="n">unavailabilities</span><span class="p">.</span><span class="k">start</span><span class="p">,</span> <span class="n">unavailabilities</span><span class="p">.</span><span class="k">end</span><span class="p">)</span> <span class="k">OVERLAPS</span>
<span class="p">(</span><span class="s1">'2016-01-31'</span><span class="p">::</span><span class="n">date</span><span class="p">,</span> <span class="s1">'2016-02-05'</span><span class="p">::</span><span class="n">date</span><span class="p">)</span></code></pre></figure>
<p>but not where to put it. After some failed experiments and more googling I came to this <a href="http://stackoverflow.com/a/6613752/1620232">stack overflow answer</a> which helped me to work out the right SQL syntax. Here’s the SQL query I ended up with:</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">rentals</span>
<span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="n">unavailabilities</span>
<span class="k">ON</span> <span class="p">(</span><span class="n">unavailabilities</span><span class="p">.</span><span class="n">rental_id</span> <span class="o">=</span> <span class="n">rentals</span><span class="p">.</span><span class="n">id</span><span class="p">)</span>
<span class="k">AND</span> <span class="p">(</span><span class="n">unavailabilities</span><span class="p">.</span><span class="k">start</span><span class="p">,</span> <span class="n">unavailabilities</span><span class="p">.</span><span class="k">end</span><span class="p">)</span>
<span class="k">OVERLAPS</span> <span class="p">(</span><span class="s1">'2016-01-31'</span><span class="p">::</span><span class="n">date</span><span class="p">,</span> <span class="s1">'2016-02-05'</span><span class="p">::</span><span class="n">date</span><span class="p">)</span>
<span class="k">WHERE</span> <span class="n">rental_id</span> <span class="k">IS</span> <span class="k">NULL</span><span class="p">;</span></code></pre></figure>
<p>Working within the Postgres console I was able to confirm that this works for my use cases.</p>
<h2 id="build-the-ecto-query-from-the-sql-query">Build the Ecto Query from the SQL Query</h2>
<p>The next step was to take the SQL query I have build and turn it into an Ecto query. Of course, I could try to use the raw SQL in a <code class="highlighter-rouge">fragment</code> but it would be nice to use a plain Ecto query if possible.</p>
<p>While doing this work I found the post <a href="http://www.glydergun.com/diving-into-ecto-part-2/">“Diving Into Ecto Part 2”</a> very helpful. As was the <a href="https://hexdocs.pm/ecto/Ecto.Query.html#join/5">Ecto documentation for <code class="highlighter-rouge">join</code></a>.</p>
<p>Just as with the raw SQL the one thing that I had a little trouble with was adding the overlaps condition in the right place. Most <code class="highlighter-rouge">join</code> examples use <code class="highlighter-rouge">assoc</code> as in this example from the docs:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">from</span> <span class="n">p</span> <span class="ow">in</span> <span class="no">Post</span><span class="p">,</span>
<span class="ss">left_join:</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">assoc</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="ss">:comments</span><span class="p">),</span>
<span class="ss">select:</span> <span class="p">{</span><span class="n">p</span><span class="p">,</span> <span class="n">c</span><span class="p">}</span></code></pre></figure>
<p>I wanted to combine the association with the <code class="highlighter-rouge">overlap</code> condition. But, this doesn’t work. Instead I had to write out the association condition by hand, which is similar to what we see in the SQL.</p>
<p>I ended up with this function to build the query:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="nv">@doc</span> <span class="sd">"""
Query rentals available on specific date range.
* `query` - Initial query to start with. Only, rentals included in this
query will be considered.
* `s` - starting date to check for availablity
* `e` - ending date to check for availablity
"""</span>
<span class="k">def</span> <span class="n">available_between</span><span class="p">(</span><span class="n">query</span> <span class="p">\\</span> <span class="no">Rental</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span> <span class="k">do</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">parse_date</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
<span class="n">e</span> <span class="o">=</span> <span class="n">parse_date</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="n">from</span> <span class="n">rental</span> <span class="ow">in</span> <span class="n">query</span><span class="p">,</span>
<span class="ss">left_join:</span> <span class="n">range</span> <span class="ow">in</span> <span class="no">Unavailability</span><span class="p">,</span>
<span class="ss">on:</span> <span class="p">(</span><span class="n">range</span><span class="o">.</span><span class="n">rental_id</span> <span class="o">==</span> <span class="n">rental</span><span class="o">.</span><span class="n">id</span><span class="p">)</span> <span class="ow">and</span> <span class="n">overlaps</span><span class="p">(</span><span class="n">range</span><span class="p">,</span> <span class="o">^</span><span class="n">s</span><span class="p">,</span> <span class="o">^</span><span class="n">e</span><span class="p">),</span>
<span class="ss">where:</span> <span class="n">is_nil</span><span class="p">(</span><span class="n">range</span><span class="o">.</span><span class="n">rental_id</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defmacro</span> <span class="n">overlaps</span><span class="p">(</span><span class="n">range</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span> <span class="k">do</span>
<span class="c1"># Uses the Postgres specific OVERLAPS function</span>
<span class="kn">quote</span> <span class="k">do</span>
<span class="n">fragment</span><span class="p">(</span><span class="sd">"</span><span class="s2">(?, ?) OVERLAPS (?, ?)"</span><span class="p">,</span>
<span class="kn">unquote</span><span class="p">(</span><span class="n">range</span><span class="p">)</span><span class="o">.</span><span class="n">start</span><span class="p">,</span> <span class="kn">unquote</span><span class="p">(</span><span class="n">range</span><span class="p">)</span><span class="o">.</span><span class="k">end</span><span class="p">,</span>
<span class="n">type</span><span class="p">(</span><span class="kn">unquote</span><span class="p">(</span><span class="n">s</span><span class="p">),</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="p">),</span> <span class="n">type</span><span class="p">(</span><span class="kn">unquote</span><span class="p">(</span><span class="n">e</span><span class="p">),</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="p">))</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>The condition <code class="highlighter-rouge">(range.rental_id == rental.id)</code> plays the same role as <code class="highlighter-rouge">assoc(rental, :unavailable_date_ranges)</code> and then I combined that with <code class="highlighter-rouge">overlaps(range, ^s, ^e)</code> to setup the left join. I think this resembles the SQL quite closely.</p>
<p>Here’s the SQL generated by Ecto as shown in the logs:</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="n">r0</span><span class="p">.</span><span class="nv">"id"</span><span class="p">,</span> <span class="n">r0</span><span class="p">.</span><span class="nv">"title"</span><span class="p">,</span> <span class="n">r0</span><span class="p">.</span><span class="nv">"description"</span><span class="p">,</span> <span class="n">r0</span><span class="p">.</span><span class="nv">"location"</span><span class="p">,</span> <span class="n">r0</span><span class="p">.</span><span class="nv">"owner_id"</span><span class="p">,</span>
<span class="n">r0</span><span class="p">.</span><span class="nv">"inserted_at"</span><span class="p">,</span> <span class="n">r0</span><span class="p">.</span><span class="nv">"updated_at"</span> <span class="k">FROM</span> <span class="nv">"rentals"</span> <span class="k">AS</span> <span class="n">r0</span>
<span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="nv">"unavailabilities"</span> <span class="k">AS</span> <span class="n">u1</span>
<span class="k">ON</span> <span class="p">(</span><span class="n">u1</span><span class="p">.</span><span class="nv">"rental_id"</span> <span class="o">=</span> <span class="n">r0</span><span class="p">.</span><span class="nv">"id"</span><span class="p">)</span>
<span class="k">AND</span> <span class="p">(</span><span class="n">u1</span><span class="p">.</span><span class="nv">"start"</span><span class="p">,</span> <span class="n">u1</span><span class="p">.</span><span class="nv">"end"</span><span class="p">)</span> <span class="k">OVERLAPS</span> <span class="p">(</span><span class="err">$</span><span class="mi">1</span><span class="p">::</span><span class="n">date</span><span class="p">,</span> <span class="err">$</span><span class="mi">2</span><span class="p">::</span><span class="n">date</span><span class="p">)</span>
<span class="k">WHERE</span> <span class="p">(</span><span class="n">u1</span><span class="p">.</span><span class="nv">"rental_id"</span> <span class="k">IS</span> <span class="k">NULL</span><span class="p">)</span> <span class="p">[</span><span class="err">{</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">31</span><span class="err">}</span><span class="p">,</span> <span class="err">{</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">5</span><span class="err">}</span><span class="p">]</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
<p>In this post we explored a bit of SQL and I learned a bit about left joins and how to use the effectively. We also looked at building up Ecto queries from SQL. I learned a lot in doing this work and I hope you were able to learn from this post.</p>
<p><a href="http://learningelixir.joekain.com/none-of-the-associations-in-ecto/">Read more...</a></p>
http://learningelixir.joekain.com/building-a-simple-reproduction-case-for-ecto-2.0-beta2016-03-08T00:00:00-08:002016-03-08T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>Recently I decided to try out Ecto 2.0 beta in one of my private projects. In doing so I ran into a small bug and decided to put together a simple case to reproduce the problem and then file an issue against Ecto. This post describes how I went about this process.</p>
<p>In my project I found that Phoenix’s <code class="highlighter-rouge">mix ecto.reset</code> command wasn’t working in that seeding the database would fail with an error like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>** (ArgumentError) no extension found for oid `1822277`
(postgrex) lib/postgrex/types.ex:298: Postgrex.Types.fetch!/2
(postgrex) lib/postgrex/types.ex:215: Postgrex.Types.encoder/2
(elixir) lib/enum.ex:1088: Enum."-map/2-lists^map/1-0-"/2
(elixir) lib/enum.ex:1088: Enum."-map/2-lists^map/1-0-"/2
(postgrex) lib/postgrex/query.ex:82: DBConnection.Query.Postgrex.Query.encoders/2
(postgrex) lib/postgrex/query.ex:43: DBConnection.Query.Postgrex.Query.describe/2
(db_connection) lib/db_connection.ex:884: DBConnection.describe_execute/5
(db_connection) lib/db_connection.ex:966: anonymous fn/4 in DBConnection.run_meter/5
(db_connection) lib/db_connection.ex:1009: DBConnection.run_begin/3
(db_connection) lib/db_connection.ex:421: DBConnection.query/4
(ecto) lib/ecto/adapters/sql.ex:380: Ecto.Adapters.SQL.struct/6
(ecto) lib/ecto/repo/schema.ex:369: Ecto.Repo.Schema.apply/5
(ecto) lib/ecto/repo/schema.ex:175: anonymous fn/11 in Ecto.Repo.Schema.do_insert/4
(ecto) lib/ecto/repo/schema.ex:108: Ecto.Repo.Schema.insert!/4
(elixir) lib/code.ex:363: Code.require_file/2
(mix) lib/mix/tasks/run.ex:68: Mix.Tasks.Run.run/1
(mix) lib/mix/task.ex:309: Mix.Task.run_alias/3
</code></pre>
</div>
<p>After experimenting a bit I concluded that the problem was related to the PostGIS extension. I set about creating a new project with the minimal set of dependencies and code that can reproduce this problem.</p>
<p>For reference, I’m using Elixir 1.2.2 and starting out with a mix.lock file like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="p">%{</span><span class="sd">"</span><span class="s2">connection"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:connection</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.0.2"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">cowboy"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:cowboy</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.0.4"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">cowlib"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:cowlib</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.0.2"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">db_connection"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:db_connection</span><span class="p">,</span> <span class="sd">"</span><span class="s2">0.2.4"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">decimal"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:decimal</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.1.1"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">ecto"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:ecto</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.1.4"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">fs"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:fs</span><span class="p">,</span> <span class="sd">"</span><span class="s2">0.9.2"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">geo"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:geo</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.0.1"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">gettext"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:gettext</span><span class="p">,</span> <span class="sd">"</span><span class="s2">0.10.0"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">phoenix"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:phoenix</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.1.4"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">phoenix_ecto"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:phoenix_ecto</span><span class="p">,</span> <span class="sd">"</span><span class="s2">2.0.1"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">phoenix_html"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:phoenix_html</span><span class="p">,</span> <span class="sd">"</span><span class="s2">2.5.0"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">phoenix_live_reload"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:phoenix_live_reload</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.0.3"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">plug"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:plug</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.1.2"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">poison"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:poison</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.5.2"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">poolboy"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:poolboy</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.5.1"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">postgrex"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:postgrex</span><span class="p">,</span> <span class="sd">"</span><span class="s2">0.11.1"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">ranch"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:ranch</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.2.1"</span><span class="p">}}</span></code></pre></figure>
<p>Then I’ll be upgrading ecto and other packages to:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gh">index aca7970..d8d5231 100644
</span><span class="gd">--- a/mix.lock
</span><span class="gi">+++ b/mix.lock
</span><span class="gu">@@ -3,15 +3,16 @@
</span> "cowlib": {:hex, :cowlib, "1.0.2"},
"db_connection": {:hex, :db_connection, "0.2.4"},
"decimal": {:hex, :decimal, "1.1.1"},
<span class="gi">+ "ecto": {:hex, :ecto, "2.0.0-beta.1"},
</span> "fs": {:hex, :fs, "0.9.2"},
"geo": {:hex, :geo, "1.0.1"},
"gettext": {:hex, :gettext, "0.10.0"},
"phoenix": {:hex, :phoenix, "1.1.4"},
<span class="gd">- "phoenix_ecto": {:hex, :phoenix_ecto, "2.0.1"},
</span><span class="gi">+ "phoenix_ecto": {:hex, :phoenix_ecto, "3.0.0-beta.2"},
</span> "phoenix_html": {:hex, :phoenix_html, "2.5.0"},
"phoenix_live_reload": {:hex, :phoenix_live_reload, "1.0.3"},
"plug": {:hex, :plug, "1.1.2"},
<span class="gd">- "poison": {:hex, :poison, "1.5.2"},
</span><span class="gi">+ "poison": {:hex, :poison, "2.1.0"},
</span> "poolboy": {:hex, :poolboy, "1.5.1"},
"postgrex": {:hex, :postgrex, "0.11.1"},
<span class="err"> "ranch": {:hex, :ranch, "1.2.1"}}</span></code></pre></figure>
<h2 id="setup-the-app">Setup the app</h2>
<p>First, let’s create a brand new Elixir / Phoenix application:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix phoenix.new oid_migration_issue
* creating oid_migration_issue/config/config.exs
* creating oid_migration_issue/config/dev.exs
* creating oid_migration_issue/config/prod.exs
* creating oid_migration_issue/config/prod.secret.exs
* creating oid_migration_issue/config/test.exs
* creating oid_migration_issue/lib/oid_migration_issue.ex
* creating oid_migration_issue/lib/oid_migration_issue/endpoint.ex
* creating oid_migration_issue/test/views/error_view_test.exs
* creating oid_migration_issue/test/support/conn_case.ex
* creating oid_migration_issue/test/support/channel_case.ex
* creating oid_migration_issue/test/test_helper.exs
* creating oid_migration_issue/web/channels/user_socket.ex
* creating oid_migration_issue/web/router.ex
* creating oid_migration_issue/web/views/error_view.ex
* creating oid_migration_issue/web/web.ex
* creating oid_migration_issue/mix.exs
* creating oid_migration_issue/README.md
* creating oid_migration_issue/web/gettext.ex
* creating oid_migration_issue/priv/gettext/errors.pot
* creating oid_migration_issue/priv/gettext/en/LC_MESSAGES/errors.po
* creating oid_migration_issue/web/views/error_helpers.ex
* creating oid_migration_issue/lib/oid_migration_issue/repo.ex
* creating oid_migration_issue/test/support/model_case.ex
* creating oid_migration_issue/priv/repo/seeds.exs
* creating oid_migration_issue/.gitignore
* creating oid_migration_issue/brunch-config.js
* creating oid_migration_issue/package.json
* creating oid_migration_issue/web/static/css/app.css
* creating oid_migration_issue/web/static/js/app.js
* creating oid_migration_issue/web/static/js/socket.js
* creating oid_migration_issue/web/static/assets/robots.txt
* creating oid_migration_issue/web/static/assets/images/phoenix.png
* creating oid_migration_issue/web/static/assets/favicon.ico
* creating oid_migration_issue/test/controllers/page_controller_test.exs
* creating oid_migration_issue/test/views/layout_view_test.exs
* creating oid_migration_issue/test/views/page_view_test.exs
* creating oid_migration_issue/web/controllers/page_controller.ex
* creating oid_migration_issue/web/templates/layout/app.html.eex
* creating oid_migration_issue/web/templates/page/index.html.eex
* creating oid_migration_issue/web/views/layout_view.ex
* creating oid_migration_issue/web/views/page_view.ex
Fetch and install dependencies? [Yn]
* running mix deps.get
* running npm install && node node_modules/brunch/bin/brunch build
We are all set! Run your Phoenix application:
$ cd oid_migration_issue
$ mix phoenix.server
You can also run your app inside IEx (Interactive Elixir) as:
$ iex -S mix phoenix.server
Before moving on, configure your database in config/dev.exs and run:
$ mix ecto.create
</code></pre>
</div>
<p>I won’t follow the instructions and run <code class="highlighter-rouge">mix ecto.create</code> just yet. Instead, I’ll add Geo as a dependency.</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gd">--- a/mix.exs
</span><span class="gi">+++ b/mix.exs
</span><span class="gu">@@ -36,7 +36,9 @@ defmodule OidMigrationIssue.Mixfile do
</span> {:phoenix_html, "~> 2.4"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
{:gettext, "~> 0.9"},
<span class="gd">- {:cowboy, "~> 1.0"}]
</span><span class="gi">+ {:cowboy, "~> 1.0"},
+ {:geo, "~> 1.0"}
+ ]
</span><span class="err"> end</span></code></pre></figure>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix deps.get
Running dependency resolution
Dependency resolution completed
geo: 1.0.1
* Getting geo (Hex package)
Checking package (https://s3.amazonaws.com/s3.hex.pm/tarballs/geo-1.0.1.tar)
Using locally cached package
</code></pre>
</div>
<p>And enable the extension for Ecto:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gd">--- a/config/dev.exs
</span><span class="gi">+++ b/config/dev.exs
</span><span class="gu">@@ -39,4 +39,5 @@ config :oid_migration_issue, OidMigrationIssue.Repo,
</span> password: "postgres",
database: "oid_migration_issue_dev",
hostname: "localhost",
<span class="gd">- pool_size: 10
</span><span class="gi">+ pool_size: 10,
</span><span class="err">+ extensions: [{Geo.PostGIS.Extension, library: Geo}]</span></code></pre></figure>
<p>We also need to enable the PostGIS extension in the database via a migration. We’ll start by generating a new migration:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix ecto.gen.migration enable_postgis
* creating priv/repo/migrations
* creating priv/repo/migrations/20160303080907_enable_postgis.exs
</code></pre>
</div>
<p>Of course, the timestamp will differ if you run this command yourself.</p>
<p>Next, we fill the following (taken from the Geo README) into the migration:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">OidMigrationIssue</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">EnablePostgis</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="k">def</span> <span class="n">up</span> <span class="k">do</span>
<span class="n">execute</span> <span class="sd">"</span><span class="s2">CREATE EXTENSION IF NOT EXISTS postgis"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">down</span> <span class="k">do</span>
<span class="n">execute</span> <span class="sd">"</span><span class="s2">DROP EXTENSION IF EXISTS postgis"</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Let’s try it out with the <code class="highlighter-rouge">ecto.setup</code> alias. This command is a convinient alias that Phoenix creates that just runs</p>
<ul>
<li><code class="highlighter-rouge">ecto.create</code> to create the database</li>
<li><code class="highlighter-rouge">ecto.migrate</code> to migrate the database</li>
<li>
<p><code class="highlighter-rouge">run priv/repo/seeds.exs</code> to seed the database.</p>
<p>$ mix ecto.setup
The database for OidMigrationIssue.Repo has been created.</p>
<p>00:10:47.066 [info] == Running OidMigrationIssue.Repo.Migrations.EnablePostgis.up/0 forward</p>
<p>00:10:47.067 [info] execute “CREATE EXTENSION IF NOT EXISTS postgis”</p>
<p>00:10:48.611 [info] == Migrated in 15.4s</p>
</li>
</ul>
<p>So far, so good.</p>
<h2 id="creating-a-postgis-model">Creating a PostGIS model</h2>
<p>Now that we have Geo and PostGIS configured we need to put them to use and create a model:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix phoenix.gen.model GeoModel geo_models geom:Geo.Point
** (Mix) Unknown type `Geo.Point` given to generator
</code></pre>
</div>
<p>Hmm, ok so we can’t use the <code class="highlighter-rouge">Geo.Point</code> type here. Instead we’ll lie and fix up the generated code later:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix phoenix.gen.model GeoModel geo_models geom:string
* creating priv/repo/migrations/20160303081329_create_geo_model.exs
* creating web/models/geo_model.ex
* creating test/models/geo_model_test.exs
Remember to update your repository by running migrations:
$ mix ecto.migrate
</code></pre>
</div>
<p>Then we’ll switch over to using <code class="highlighter-rouge">Geo.Point</code> and <code class="highlighter-rouge">:geometry</code>:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gh">diff --git a/priv/repo/migrations/20160303081329_create_geo_model.exs b/priv/repo/migrations/20160303081329_create_geo_model.exs
index 62db09b..abc0c55 100644
</span><span class="gd">--- a/priv/repo/migrations/20160303081329_create_geo_model.exs
</span><span class="gi">+++ b/priv/repo/migrations/20160303081329_create_geo_model.exs
</span><span class="gu">@@ -3,7 +3,7 @@ defmodule OidMigrationIssue.Repo.Migrations.CreateGeoModel do
</span>
def change do
create table(:geo_models) do
<span class="gd">- add :geom, :string
</span><span class="gi">+ add :geom, :geometry
</span>
timestamps
end
<span class="gh">diff --git a/test/models/geo_model_test.exs b/test/models/geo_model_test.exs
index d6f380b..29cb661 100644
</span><span class="gd">--- a/test/models/geo_model_test.exs
</span><span class="gi">+++ b/test/models/geo_model_test.exs
</span><span class="gu">@@ -3,7 +3,7 @@ defmodule OidMigrationIssue.GeoModelTest do
</span>
alias OidMigrationIssue.GeoModel
<span class="gd">- @valid_attrs %{geom: "some content"}
</span><span class="gi">+ @valid_attrs %{geom: %Geo.Point{coordinates: {30, -90}, srid: 4326}}
</span> @invalid_attrs %{}
test "changeset with valid attributes" do
<span class="gh">diff --git a/web/models/geo_model.ex b/web/models/geo_model.ex
index f0eeeff..46dfe49 100644
</span><span class="gd">--- a/web/models/geo_model.ex
</span><span class="gi">+++ b/web/models/geo_model.ex
</span><span class="gu">@@ -2,7 +2,7 @@ defmodule OidMigrationIssue.GeoModel do
</span> use OidMigrationIssue.Web, :model
schema "geo_models" do
<span class="gd">- field :geom, :string
</span><span class="gi">+ field :geom, Geo.Point
</span>
timestamps
<span class="err"> end</span></code></pre></figure>
<h2 id="adding-seeds">Adding Seeds</h2>
<p>Let’s just write a simple seeds script to seed a point into our database. Here’s my seeds.exs script:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">alias</span> <span class="no">OidMigrationIssue</span><span class="o">.</span><span class="p">{</span><span class="no">Repo</span><span class="p">,</span> <span class="no">GeoModel</span><span class="p">}</span>
<span class="p">%</span><span class="no">GeoModel</span><span class="p">{}</span>
<span class="o">|></span> <span class="no">GeoModel</span><span class="o">.</span><span class="n">changeset</span><span class="p">(%{</span><span class="ss">geom:</span> <span class="p">%</span><span class="no">Geo</span><span class="o">.</span><span class="no">Point</span><span class="p">{</span><span class="ss">coordinates:</span> <span class="p">{</span><span class="m">30</span><span class="p">,</span> <span class="o">-</span><span class="m">90</span><span class="p">},</span> <span class="ss">srid:</span> <span class="m">4326</span><span class="p">}})</span>
<span class="o">|></span> <span class="no">Repo</span><span class="o">.</span><span class="n">insert!</span></code></pre></figure>
<p>Then, we can use <code class="highlighter-rouge">ecto.reset</code> to reset our database using the new seeds:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix ecto.reset
The database for OidMigrationIssue.Repo has been dropped.
The database for OidMigrationIssue.Repo has been created.
00:21:44.809 [info] == Running OidMigrationIssue.Repo.Migrations.EnablePostgis.up/0 forward
00:21:44.811 [info] execute "CREATE EXTENSION IF NOT EXISTS postgis"
00:21:45.941 [info] == Migrated in 11.2s
00:21:45.996 [info] == Running OidMigrationIssue.Repo.Migrations.CreateGeoModel.change/0 forward
00:21:45.996 [info] create table geo_models
00:21:46.006 [info] == Migrated in 0.1s
[debug] INSERT INTO "geo_models" ("inserted_at", "updated_at", "geom") VALUES ($1, $2, $3) RETURNING "id" [{ {2016, 3, 3}, {8, 21, 46, 0} }, { {2016, 3, 3}, {8, 21, 46, 0} }, %Geo.Point{coordinates: {30, -90}, srid: 4326}] OK query=43.3ms
</code></pre>
</div>
<p>Good, everything seems to be working fine.</p>
<h2 id="updating-to-ecto-20-beta">Updating to Ecto 2.0 beta</h2>
<p>The next step is to upgrade to Ecto 2.0 beta. We’ll follow the instructions from http://blog.plataformatec.com.br/2016/02/ecto-2-0-0-beta-0-is-out/. Fortunately, we don’t have that much to do since we’ve written so little code.</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gh">diff --git a/mix.exs b/mix.exs
index 58cda12..f2a3e9e 100644
</span><span class="gd">--- a/mix.exs
</span><span class="gi">+++ b/mix.exs
</span><span class="gu">@@ -32,7 +32,7 @@ defmodule OidMigrationIssue.Mixfile do
</span> defp deps do
[{:phoenix, "~> 1.1.4"},
{:postgrex, ">= 0.0.0"},
<span class="gd">- {:phoenix_ecto, "~> 2.0"},
</span><span class="gi">+ {:phoenix_ecto, "~> 3.0.0-beta"},
</span> {:phoenix_html, "~> 2.4"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
{:gettext, "~> 0.9"},
<span class="gh">diff --git a/test/support/channel_case.ex b/test/support/channel_case.ex
index ca65c92..bd242ec 100644
</span><span class="gd">--- a/test/support/channel_case.ex
</span><span class="gi">+++ b/test/support/channel_case.ex
</span><span class="gu">@@ -32,10 +32,7 @@ defmodule OidMigrationIssue.ChannelCase do
</span> end
setup tags do
<span class="gd">- unless tags[:async] do
- Ecto.Adapters.SQL.restart_test_transaction(OidMigrationIssue.Repo, [])
- end
-
</span><span class="gi">+ :ok = Ecto.Adapters.SQL.Sandbox.checkout(Demo.Repo)
</span> :ok
end
end
<span class="gh">diff --git a/test/support/conn_case.ex b/test/support/conn_case.ex
index 8affedf..67dd527 100644
</span><span class="gd">--- a/test/support/conn_case.ex
</span><span class="gi">+++ b/test/support/conn_case.ex
</span><span class="gu">@@ -33,10 +33,7 @@ defmodule OidMigrationIssue.ConnCase do
</span> end
setup tags do
<span class="gd">- unless tags[:async] do
- Ecto.Adapters.SQL.restart_test_transaction(OidMigrationIssue.Repo, [])
- end
-
</span><span class="gi">+ :ok = Ecto.Adapters.SQL.Sandbox.checkout(Demo.Repo)
</span> {:ok, conn: Phoenix.ConnTest.conn()}
end
end
<span class="gh">diff --git a/test/support/model_case.ex b/test/support/model_case.ex
index c3b0e07..6db2da6 100644
</span><span class="gd">--- a/test/support/model_case.ex
</span><span class="gi">+++ b/test/support/model_case.ex
</span><span class="gu">@@ -26,10 +26,7 @@ defmodule OidMigrationIssue.ModelCase do
</span> end
setup tags do
<span class="gd">- unless tags[:async] do
- Ecto.Adapters.SQL.restart_test_transaction(OidMigrationIssue.Repo, [])
- end
-
</span><span class="gi">+ :ok = Ecto.Adapters.SQL.Sandbox.checkout(Demo.Repo)
</span> :ok
end
<span class="gh">diff --git a/test/test_helper.exs b/test/test_helper.exs
index 973f1a8..6892146 100644
</span><span class="gd">--- a/test/test_helper.exs
</span><span class="gi">+++ b/test/test_helper.exs
</span><span class="gu">@@ -2,5 +2,4 @@ ExUnit.start
</span>
Mix.Task.run "ecto.create", ~w(-r OidMigrationIssue.Repo --quiet)
Mix.Task.run "ecto.migrate", ~w(-r OidMigrationIssue.Repo --quiet)
<span class="gd">-Ecto.Adapters.SQL.begin_test_transaction(OidMigrationIssue.Repo)
-
</span><span class="err">+Ecto.Adapters.SQL.Sandbox.mode(OidMigrationIssue.Repo, :manual)</span></code></pre></figure>
<p>Now, to install the deps:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix deps.get
Running dependency resolution
Conflict on ecto
mix.lock: 1.1.4
phoenix_ecto from 3.0.0-beta.0 to 3.0.0-beta.2: ~> 2.0-beta
** (Mix) Hex dependency resolution failed, relax the version requirements
or unlock dependencies
</code></pre>
</div>
<p>We need to follow the instructions here and unlock ecto:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix deps.unlock ecto
$ mix deps.get
Running dependency resolution
Conflict on ecto 2.0.0-beta.0, 2.0.0-beta.1
geo 1.0.1: ~> 1.1
phoenix_ecto from 3.0.0-beta.0 to 3.0.0-beta.2: ~> 2.0-beta
** (Mix) Hex dependency resolution failed, relax the version requirements
or unlock dependencies
</code></pre>
</div>
<p>Ah, so Geo says it depends on Ecto 1.1. I’ve tested this before and I think Geo can work fine with Ecto 2.0 so let’s override it:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gh">diff --git a/mix.exs b/mix.exs
index f2a3e9e..1772008 100644
</span><span class="gd">--- a/mix.exs
</span><span class="gi">+++ b/mix.exs
</span><span class="gu">@@ -35,6 +35,7 @@ defmodule OidMigrationIssue.Mixfile do
</span> {:phoenix_ecto, "~> 3.0.0-beta"},
{:phoenix_html, "~> 2.4"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
<span class="gi">+ {:ecto, "~> 2.0-beta", override: true},
</span> {:gettext, "~> 0.9"},
{:cowboy, "~> 1.0"},
<span class="err"> {:geo, "~> 1.0"}</span></code></pre></figure>
<p>And now we can install the deps we want:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix deps.unlock --all
$ mix deps.get
Running dependency resolution
Dependency resolution completed
connection: 1.0.2
cowboy: 1.0.4
cowlib: 1.0.2
db_connection: 0.2.4
decimal: 1.1.1
ecto: 2.0.0-beta.1
fs: 0.9.2
geo: 1.0.1
gettext: 0.10.0
phoenix: 1.1.4
phoenix_ecto: 3.0.0-beta.2
phoenix_html: 2.5.0
phoenix_live_reload: 1.0.3
plug: 1.1.2
poison: 2.1.0
poolboy: 1.5.1
postgrex: 0.11.1
ranch: 1.2.1
* Updating poison (Hex package)
Checking package (https://s3.amazonaws.com/s3.hex.pm/tarballs/poison-2.1.0.tar)
Using locally cached package
</code></pre>
</div>
<h2 id="ecto-reset">Ecto reset</h2>
<p>Now, let’s retry our <code class="highlighter-rouge">ecto.reset</code> command again:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix ecto.reset
The database for OidMigrationIssue.Repo has been dropped.
The database for OidMigrationIssue.Repo has been created.
00:34:57.457 [info] == Running OidMigrationIssue.Repo.Migrations.EnablePostgis.up/0 forward
00:34:57.459 [info] execute "CREATE EXTENSION IF NOT EXISTS postgis"
00:34:58.653 [info] == Migrated in 11.9s
00:34:58.710 [info] == Running OidMigrationIssue.Repo.Migrations.CreateGeoModel.change/0 forward
00:34:58.710 [info] create table geo_models
00:34:58.715 [info] == Migrated in 0.0s
** (ArgumentError) no extension found for oid `1822277`
(postgrex) lib/postgrex/types.ex:298: Postgrex.Types.fetch!/2
(postgrex) lib/postgrex/types.ex:215: Postgrex.Types.encoder/2
(elixir) lib/enum.ex:1088: Enum."-map/2-lists^map/1-0-"/2
(elixir) lib/enum.ex:1088: Enum."-map/2-lists^map/1-0-"/2
(postgrex) lib/postgrex/query.ex:82: DBConnection.Query.Postgrex.Query.encoders/2
(postgrex) lib/postgrex/query.ex:43: DBConnection.Query.Postgrex.Query.describe/2
(db_connection) lib/db_connection.ex:884: DBConnection.describe_execute/5
(db_connection) lib/db_connection.ex:966: anonymous fn/4 in DBConnection.run_meter/5
(db_connection) lib/db_connection.ex:1009: DBConnection.run_begin/3
(db_connection) lib/db_connection.ex:421: DBConnection.query/4
(ecto) lib/ecto/adapters/sql.ex:380: Ecto.Adapters.SQL.struct/6
(ecto) lib/ecto/repo/schema.ex:369: Ecto.Repo.Schema.apply/5
(ecto) lib/ecto/repo/schema.ex:175: anonymous fn/11 in Ecto.Repo.Schema.do_insert/4
(ecto) lib/ecto/repo/schema.ex:108: Ecto.Repo.Schema.insert!/4
(elixir) lib/code.ex:363: Code.require_file/2
(mix) lib/mix/tasks/run.ex:68: Mix.Tasks.Run.run/1
(mix) lib/mix/task.ex:309: Mix.Task.run_alias/3
</code></pre>
</div>
<p>Ok, good. This is the failure we expected. Our reproduction case is doing its job. Let’s confirm it’s behaving the way I claim it is.</p>
<p>Note that if I remove the seed command from the aliases like this:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gh">diff --git a/mix.exs b/mix.exs
index 1772008..ccd92b7 100644
</span><span class="gd">--- a/mix.exs
</span><span class="gi">+++ b/mix.exs
</span><span class="gu">@@ -49,7 +49,7 @@ defmodule OidMigrationIssue.Mixfile do
</span> #
# See the documentation for `Mix` for more info on aliases.
defp aliases do
<span class="gd">- ["ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
</span><span class="gi">+ ["ecto.setup": ["ecto.create", "ecto.migrate"],
</span> "ecto.reset": ["ecto.drop", "ecto.setup"]]
end
<span class="err"> end</span></code></pre></figure>
<p>and then run reset and seed as two commands then everything works as it should:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix ecto.reset && mix run priv/repo/seeds.exs
The database for OidMigrationIssue.Repo has been dropped.
The database for OidMigrationIssue.Repo has been created.
09:10:52.465 [info] == Running OidMigrationIssue.Repo.Migrations.EnablePostgis.up/0 forward
09:10:52.467 [info] execute "CREATE EXTENSION IF NOT EXISTS postgis"
09:10:53.649 [info] == Migrated in 11.8s
09:10:53.702 [info] == Running OidMigrationIssue.Repo.Migrations.CreateGeoModel.change/0 forward
09:10:53.702 [info] create table geo_models
09:10:53.707 [info] == Migrated in 0.0s
[debug] INSERT INTO "geo_models" ("inserted_at","updated_at","geom") VALUES ($1,$2,$3) RETURNING "id" [{ {2016, 3, 3}, {17, 10, 54, 0} }, { {2016, 3, 3}, {17, 10, 54, 0} }, %Geo.Point{coordinates: {30, -90}, srid: 4326}] OK query=6.7ms queue=48.7ms
</code></pre>
</div>
<p>So this appears to be a problem running the commands within a single mix invocation.</p>
<h2 id="look-over-the-existing-issues">Look over the existing issues</h2>
<p>Before I go ahead and file an issue against Ecto I should check to see if there is one already filed. Searching, I see <a href="https://github.com/elixir-lang/ecto/issues/1080">Reload TYPE ENUM (Postgre) before migration</a> which has been fixed. But, reading through the issue, it sounds like it describes a very similar issue that would come up when migrating a database due to the Postgres types cache. I wonder if our issue is due to the same root cause? That is, the problem has been fixed for migrations but the problem affects seeds as well.</p>
<p>I can try adding some code to my seeds based on the fix for the above issue. Based on my reading of the fixes I need to add:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gh">diff --git a/priv/repo/seeds.exs b/priv/repo/seeds.exs
index 13152f0..dd2b82b 100644
</span><span class="gd">--- a/priv/repo/seeds.exs
</span><span class="gi">+++ b/priv/repo/seeds.exs
</span><span class="gu">@@ -12,6 +12,9 @@
</span>
alias OidMigrationIssue.{Repo, GeoModel}
<span class="gi">+{:ok, _} = Application.ensure_all_started(:ecto)
+{:ok, _} = Application.ensure_all_started(:postgrex)
+
</span> %GeoModel{}
|> GeoModel.changeset(%{geom: %Geo.Point{coordinates: {30, -90}, srid: 4326}})
<span class="err"> |> Repo.insert!</span></code></pre></figure>
<p>But running this version I still get the same error.</p>
<h2 id="try-out-ecto-master">Try out Ecto master</h2>
<p>We should try out the master branch of Ecto to see if the issue has been fixed there but not yet published in a release. To do this we make this change to mix.exs:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gh">diff --git a/mix.exs b/mix.exs
index 1772008..5c0827b 100644
</span><span class="gd">--- a/mix.exs
</span><span class="gi">+++ b/mix.exs
</span><span class="gu">@@ -35,7 +35,7 @@ defmodule OidMigrationIssue.Mixfile do
</span> {:phoenix_ecto, "~> 3.0.0-beta"},
{:phoenix_html, "~> 2.4"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
<span class="gd">- {:ecto, "~> 2.0-beta", override: true},
</span><span class="gi">+ {:ecto, github: "elixir-lang/ecto", branch: "master", override: true},
</span> {:gettext, "~> 0.9"},
{:cowboy, "~> 1.0"},
<span class="err"> {:geo, "~> 1.0"}</span></code></pre></figure>
<p>Then update the deps</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix deps.get
* Getting ecto (https://github.com/elixir-lang/ecto.git)
Cloning into '/Users/jkain/Documents/Projects/elixir/oid_migration_issue/deps/ecto'...
remote: Counting objects: 25829, done.
remote: Compressing objects: 100% (232/232), done.
remote: Total 25829 (delta 110), reused 2 (delta 2), pack-reused 25594
Receiving objects: 100% (25829/25829), 6.58 MiB | 2.59 MiB/s, done.
Resolving deltas: 100% (14906/14906), done.
Checking connectivity... done.
Running dependency resolution
</code></pre>
</div>
<p>I see the same problem:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix ecto.reset
The database for OidMigrationIssue.Repo has been dropped.
The database for OidMigrationIssue.Repo has been created.
09:20:59.536 [info] == Running OidMigrationIssue.Repo.Migrations.EnablePostgis.up/0 forward
09:20:59.537 [info] execute "CREATE EXTENSION IF NOT EXISTS postgis"
09:21:00.774 [info] == Migrated in 12.3s
09:21:00.817 [info] == Running OidMigrationIssue.Repo.Migrations.CreateGeoModel.change/0 forward
09:21:00.817 [info] create table geo_models
09:21:00.822 [info] == Migrated in 0.0s
** (ArgumentError) no extension found for oid `1841936`
(postgrex) lib/postgrex/types.ex:298: Postgrex.Types.fetch!/2
(postgrex) lib/postgrex/types.ex:215: Postgrex.Types.encoder/2
(elixir) lib/enum.ex:1088: Enum."-map/2-lists^map/1-0-"/2
(elixir) lib/enum.ex:1088: Enum."-map/2-lists^map/1-0-"/2
(postgrex) lib/postgrex/query.ex:82: DBConnection.Query.Postgrex.Query.encoders/2
(postgrex) lib/postgrex/query.ex:43: DBConnection.Query.Postgrex.Query.describe/2
(db_connection) lib/db_connection.ex:884: DBConnection.describe_execute/5
(db_connection) lib/db_connection.ex:966: anonymous fn/4 in DBConnection.run_meter/5
(db_connection) lib/db_connection.ex:1009: DBConnection.run_begin/3
(db_connection) lib/db_connection.ex:421: DBConnection.query/4
(ecto) lib/ecto/adapters/sql.ex:387: Ecto.Adapters.SQL.struct/6
(ecto) lib/ecto/repo/schema.ex:369: Ecto.Repo.Schema.apply/5
(ecto) lib/ecto/repo/schema.ex:175: anonymous fn/11 in Ecto.Repo.Schema.do_insert/4
(ecto) lib/ecto/repo/schema.ex:108: Ecto.Repo.Schema.insert!/4
(elixir) lib/code.ex:363: Code.require_file/2
(mix) lib/mix/tasks/run.ex:68: Mix.Tasks.Run.run/1
(mix) lib/mix/task.ex:309: Mix.Task.run_alias/3
</code></pre>
</div>
<h2 id="file-an-issue-and-see-it-fixed">File an Issue and See it Fixed</h2>
<p>I filed <a href="https://github.com/elixir-lang/ecto/issues/1289">a new issue</a> and filled in the template and a link to <a href="https://github.com/joekain/oid_migration_issue">the repo for this simple reproducer</a>.</p>
<p>The issue was fixed in under an hour and half. Here’s the <a href="https://github.com/elixir-lang/ecto/commit/df13b1c64f8edd128cec1316336b20f3153eafa3">commit</a>.</p>
<p>The fix is in master, let’s try it out. We already have the change to fetch Ecto from master. I just need to unlock Ecto and update to the newest master. Then</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix ecto.reset
The database for OidMigrationIssue.Repo has been dropped.
The database for OidMigrationIssue.Repo has been created.
07:37:25.415 [info] == Running OidMigrationIssue.Repo.Migrations.EnablePostgis.up/0 forward
07:37:25.417 [info] execute "CREATE EXTENSION IF NOT EXISTS postgis"
07:37:26.941 [info] == Migrated in 15.2s
07:37:26.994 [info] == Running OidMigrationIssue.Repo.Migrations.CreateGeoModel.change/0 forward
07:37:26.994 [info] create table geo_models
07:37:26.999 [info] == Migrated in 0.0s
[debug] INSERT INTO "geo_models" ("inserted_at","updated_at","geom") VALUES ($1,$2,$3) RETURNING "id" [{ {2016, 3, 5}, {15, 37, 27, 0} }, { {2016, 3, 5}, {15, 37, 27, 0} }, %Geo.Point{coordinates: {30, -90}, srid: 4326}] OK query=7.0ms queue=27.3ms
</code></pre>
</div>
<p>It works! Thanks José and Ecto team!</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope you enjoyed this post on my experiences with Ecto 2.0-beta and building a simple reproduction case. I know it was a good oppourtunity for me to help out with the testing of Ecto 2.0.</p>
<p><a href="http://learningelixir.joekain.com/building-a-simple-reproduction-case-for-ecto-2.0-beta/">Read more...</a></p>
http://learningelixir.joekain.com/using-genevent-to-notify-a-channel2016-02-24T00:00:00-08:002016-02-24T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>Last week I wrote about <a href="/pushing-model-changes-to-a-phoenix-channel/">publishing model data over a Phoenix Channel</a>. My partner on the project I’m working suggested that we could use <code class="highlighter-rouge">GenEvent</code> to manage the model change notifications. The rest of this post will develop that idea.</p>
<p>GenEvent is a mechanism for managing event notifications. It consists of an event manager and one or more event handlers. We can try to use GenEvent by sending notification of model changes to the event manager. Then, we need an event handler which will be called for each event. In our handler we can respond to the event by publishing the data on the channel.</p>
<div style="text-align:center">
<p><img src="http://learningelixir.joekain.com/images/event-arch.png" alt="Event Architecture" title="Diagram of controller, event manager, and channel." /></p>
</div>
<p>This has the nice property that the source of events (the controller in our case) and the receiver(s) of the events (the channel in our case) don’t need to know about each other. Each module only needs to know about event manager. For this small example it isn’t a big deal but imagine that we had a larger application with multiple sources and receivers. We certainly don’t want our controller(s) to grow in complexity as the number of event receivers increases.</p>
<p>That’s the plan anyway, let’s start trying to put it together.</p>
<h2 id="genevent-and-phoenixchannel">GenEvent and Phoenix.Channel</h2>
<p>Here’s my first question: can I make my channel both a channel and a GenEvent? I don’t know the answer to this so let’s try it out.</p>
<p>First we add</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="kn">use</span> <span class="no">GenEvent</span></code></pre></figure>
<p>to <code class="highlighter-rouge">PlayChannel.ToyChannel</code> to implement the behaviour. A quick check of the show page makes it look like it’s still working. At this point I sort of wish I had written up some tests.</p>
<p>I’ll go back and fix any broken generated tests and write a few more.</p>
<h2 id="channel-tests">Channel tests</h2>
<p>I’ll back out the change to <code class="highlighter-rouge">use GenEvent</code> get the tests passing and then add it back in. Looks like only (and all) the toy_channel_tests are failing:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> 1) test ping replies with status ok (PlayChannel.ToyChannelTest)
test/channels/toy_channel_test.exs:14
** (EXIT from #PID<0.2791.0>) an exception was raised:
** (Ecto.CastError) deps/ecto/lib/ecto/repo/queryable.ex:188: value `"lobby"` in `where` cannot be cast to type :id in query:
from t in PlayChannel.Toy,
where: t.id == ^"lobby"
Error when casting value to `PlayChannel.Toy.id`
(elixir) lib/enum.ex:1473: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:1151: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
(ecto) lib/ecto/repo/queryable.ex:91: Ecto.Repo.Queryable.execute/5
(ecto) lib/ecto/repo/queryable.ex:15: Ecto.Repo.Queryable.all/4
(ecto) lib/ecto/repo/queryable.ex:44: Ecto.Repo.Queryable.one/4
(play_channel) web/channels/toy_channel.ex:5: PlayChannel.ToyChannel.join/3
(phoenix) lib/phoenix/channel/server.ex:168: Phoenix.Channel.Server.init/1
(stdlib) gen_server.erl:328: :gen_server.init_it/6
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
2) test broadcasts are pushed to the client (PlayChannel.ToyChannelTest)
test/channels/toy_channel_test.exs:24
** (EXIT from #PID<0.2793.0>) an exception was raised:
** (Ecto.CastError) deps/ecto/lib/ecto/repo/queryable.ex:188: value `"lobby"` in `where` cannot be cast to type :id in query:
from t in PlayChannel.Toy,
where: t.id == ^"lobby"
Error when casting value to `PlayChannel.Toy.id`
(elixir) lib/enum.ex:1473: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:1151: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
(ecto) lib/ecto/repo/queryable.ex:91: Ecto.Repo.Queryable.execute/5
(ecto) lib/ecto/repo/queryable.ex:15: Ecto.Repo.Queryable.all/4
(ecto) lib/ecto/repo/queryable.ex:44: Ecto.Repo.Queryable.one/4
(play_channel) web/channels/toy_channel.ex:5: PlayChannel.ToyChannel.join/3
(phoenix) lib/phoenix/channel/server.ex:168: Phoenix.Channel.Server.init/1
(stdlib) gen_server.erl:328: :gen_server.init_it/6
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
3) test shout broadcasts to toys:lobby (PlayChannel.ToyChannelTest)
test/channels/toy_channel_test.exs:19
** (EXIT from #PID<0.2795.0>) an exception was raised:
** (Ecto.CastError) deps/ecto/lib/ecto/repo/queryable.ex:188: value `"lobby"` in `where` cannot be cast to type :id in query:
from t in PlayChannel.Toy,
where: t.id == ^"lobby"
Error when casting value to `PlayChannel.Toy.id`
(elixir) lib/enum.ex:1473: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:1151: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
(ecto) lib/ecto/repo/queryable.ex:91: Ecto.Repo.Queryable.execute/5
(ecto) lib/ecto/repo/queryable.ex:15: Ecto.Repo.Queryable.all/4
(ecto) lib/ecto/repo/queryable.ex:44: Ecto.Repo.Queryable.one/4
(play_channel) web/channels/toy_channel.ex:5: PlayChannel.ToyChannel.join/3
(phoenix) lib/phoenix/channel/server.ex:168: Phoenix.Channel.Server.init/1
(stdlib) gen_server.erl:328: :gen_server.init_it/6
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
</code></pre>
</div>
<p>The “shout” and “ping” tests need to be removed because I removed support for these messages from the channel. That leaves us with the broadcast test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">broadcasts are pushed to the client"</span><span class="p">,</span> <span class="p">%{</span><span class="ss">socket:</span> <span class="n">socket</span><span class="p">}</span> <span class="k">do</span>
<span class="n">broadcast_from!</span> <span class="n">socket</span><span class="p">,</span> <span class="sd">"</span><span class="s2">broadcast"</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">some"</span> <span class="o">=></span> <span class="sd">"</span><span class="s2">data"</span><span class="p">}</span>
<span class="n">assert_push</span> <span class="sd">"</span><span class="s2">broadcast"</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">some"</span> <span class="o">=></span> <span class="sd">"</span><span class="s2">data"</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>And this fails because the generated test tries to join “toys:lobby” which we don’t recognize as a room. Fixing this a little difficult, I need a toy to be present in the database. I’ll have to update the setup funciton:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="nv">@valid_toy</span> <span class="p">%{</span><span class="ss">age:</span> <span class="m">42</span><span class="p">,</span> <span class="ss">color:</span> <span class="sd">"</span><span class="s2">some content"</span><span class="p">,</span> <span class="ss">name:</span> <span class="sd">"</span><span class="s2">some content"</span><span class="p">}</span>
<span class="n">setup</span> <span class="k">do</span>
<span class="n">toy</span> <span class="o">=</span> <span class="no">Toy</span><span class="o">.</span><span class="n">changeset</span><span class="p">(%</span><span class="no">Toy</span><span class="p">{},</span> <span class="nv">@valid_toy</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Repo</span><span class="o">.</span><span class="n">insert!</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">socket</span><span class="p">}</span> <span class="o">=</span>
<span class="n">socket</span><span class="p">(</span><span class="sd">"</span><span class="s2">user_id"</span><span class="p">,</span> <span class="p">%{</span><span class="ss">some:</span> <span class="ss">:assign</span><span class="p">})</span>
<span class="o">|></span> <span class="n">subscribe_and_join</span><span class="p">(</span><span class="no">ToyChannel</span><span class="p">,</span> <span class="sd">"</span><span class="s2">toys:</span><span class="si">#{</span><span class="n">toy</span><span class="o">.</span><span class="n">id</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="ss">socket:</span> <span class="n">socket</span><span class="p">,</span> <span class="ss">payload:</span> <span class="n">payload</span><span class="p">,</span> <span class="ss">toy:</span> <span class="n">toy</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>Here we create a new toy and insert it into the database. Then we join the associated channel. I’ve also captured the payload from the reply and made it available for the tests. Futhermore, I’ve made the toy available to the test. I didn’t need these in the test right now but I think I will soon. With this change the test passes.</p>
<p>Now I’ll validate that we get the messages we expect and we’ll use the <code class="highlighter-rouge">payload</code> and <code class="highlighter-rouge">toy</code> to do it.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">join delivers the original model"</span><span class="p">,</span> <span class="p">%{</span><span class="ss">payload:</span> <span class="n">payload</span><span class="p">,</span> <span class="ss">toy:</span> <span class="n">toy</span><span class="p">}</span> <span class="k">do</span>
<span class="n">assert</span> <span class="n">payload</span> <span class="o">==</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">age"</span> <span class="o">=></span> <span class="m">42</span><span class="p">,</span> <span class="sd">"</span><span class="s2">color"</span> <span class="o">=></span> <span class="sd">"</span><span class="s2">some content"</span><span class="p">,</span>
<span class="sd">"</span><span class="s2">name"</span> <span class="o">=></span> <span class="sd">"</span><span class="s2">some content"</span><span class="p">,</span> <span class="sd">"</span><span class="s2">id"</span> <span class="o">=></span> <span class="n">toy</span><span class="o">.</span><span class="n">id</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>This test passes as well.</p>
<p>Finally, we need to perform an update to the toy and verify that there is a broadcast. This test passes:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">broadcast_change tiggers a broadcast"</span><span class="p">,</span> <span class="p">%{</span><span class="ss">toy:</span> <span class="n">toy</span><span class="p">}</span> <span class="k">do</span>
<span class="no">Toy</span><span class="o">.</span><span class="n">changeset</span><span class="p">(</span><span class="n">toy</span><span class="p">,</span> <span class="p">%{</span><span class="ss">age:</span> <span class="m">3</span><span class="p">})</span>
<span class="o">|></span> <span class="no">Repo</span><span class="o">.</span><span class="n">update!</span>
<span class="o">|></span> <span class="no">ToyChannel</span><span class="o">.</span><span class="n">broadcast_change</span>
<span class="n">assert_broadcast</span> <span class="sd">"</span><span class="s2">change"</span><span class="p">,</span> <span class="n">_</span>
<span class="k">end</span></code></pre></figure>
<h2 id="building-a-genevent">Building a GenEvent</h2>
<p>Now that I have some tests I’ll add <code class="highlighter-rouge">use GenEvent</code> line to <code class="highlighter-rouge">PlayChannel.ToyChannel</code>. The tests still pass but, I get some warnings:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>web/channels/toy_channel.ex:1: warning: conflicting behaviours - callback code_change/3 required by both 'gen_event' and ''Elixir.Phoenix.Channel'' (line 1)
web/channels/toy_channel.ex:1: warning: conflicting behaviours - callback handle_info/2 required by both 'gen_event' and ''Elixir.Phoenix.Channel'' (line 1)
web/channels/toy_channel.ex:1: warning: conflicting behaviours - callback terminate/2 required by both 'gen_event' and ''Elixir.Phoenix.Channel'' (line 1)
</code></pre>
</div>
<p>This makes sense, both Channel and GenEvent have callbacks <code class="highlighter-rouge">code_change</code>, <code class="highlighter-rouge">handle_info</code>, and <code class="highlighter-rouge">terminate</code>. As both are OTP compliant I suspect the callbacks have the same purpose in both behaviours and could be compatible. But, this seems like an untravelled path and may not be a great idea.</p>
<p>Instead of having my channel act as both a channel and a GenEvent I’ll create a seperate GenEvent module to forward the messages on to the channel. I’ll start with this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandler</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenEvent</span>
<span class="k">end</span></code></pre></figure>
<p>Now, what do I need to fill in? Reading the documentation for GenEvent I see that I need a <code class="highlighter-rouge">handle_event</code> function. It takes an event and current state as arguments and needs to return <code class="highlighter-rouge"><span class="p">{</span><span class="err">:ok,</span><span class="w"> </span><span class="err">newstate</span><span class="p">}</span></code>. <code class="highlighter-rouge">newstate</code>? Do I need state in this handler? I don’t think so. So let’s do this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">handle_event</span><span class="p">({</span><span class="ss">:update</span><span class="p">,</span> <span class="n">toy</span><span class="p">},</span> <span class="n">_</span><span class="p">)</span> <span class="k">do</span>
<span class="no">PlayChannel</span><span class="o">.</span><span class="no">ToyChannel</span><span class="o">.</span><span class="n">broadcast_change</span><span class="p">(</span><span class="n">toy</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">nil</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>Our <code class="highlighter-rouge">handle_event</code> handles <code class="highlighter-rouge">:update</code> messages that include the <code class="highlighter-rouge">toy</code> that was updated. Then they forward <code class="highlighter-rouge">toy</code> on to <code class="highlighter-rouge">PlayChannel.ToyChannel.broadcast_change</code> so that the channel can respond. We return <code class="highlighter-rouge">:ok</code> with <code class="highlighter-rouge">nil</code> as our new state.</p>
<p>Now I think we have a functioning GenEvent handler. I have no tests to prove it but it looks good. The question now is: how do we get events into it?</p>
<p>We need to start a GenEvent event manager using <code class="highlighter-rouge">GenEvent.start_link/1</code> and have the controller send events to it instead of sending them directly to the channel. We also need to register <code class="highlighter-rouge">PlayChannel.Toy.UpdateEventHandler</code> as a handler.</p>
<p>Also, I don’t just want to “start a GenEvent event manager” I want to add it to my supervision tree and name it. I do that by adding a worker entry in my child spec for the <code class="highlighter-rouge">PlayChannel</code> application:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="err">@@</span> <span class="o">-</span><span class="m">13</span><span class="p">,</span><span class="m">6</span> <span class="o">+</span><span class="m">13</span><span class="p">,</span><span class="m">7</span> <span class="err">@@</span> <span class="k">defmodule</span> <span class="no">PlayChannel</span> <span class="k">do</span>
<span class="n">supervisor</span><span class="p">(</span><span class="no">PlayChannel</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span> <span class="p">[]),</span>
<span class="c1"># Here you could define other workers and supervisors as children</span>
<span class="c1"># worker(PlayChannel.Worker, [arg1, arg2, arg3]),</span>
<span class="o">+</span> <span class="n">worker</span><span class="p">(</span><span class="no">GenEvent</span><span class="p">,</span> <span class="p">[[</span><span class="ss">name:</span> <span class="ss">:toy_event_manager</span><span class="p">]])</span>
<span class="p">]</span>
<span class="c1"># See http://elixir-lang.org/docs/stable/elixir/Supervisor.html</span></code></pre></figure>
<p>Next we need to register <code class="highlighter-rouge">PlayChannel.Toy.UpdateEventHandler</code> as a handler. This is easy to do by calling <code class="highlighter-rouge">GenEvent.add_handler/2</code> but the question is where to make the call?</p>
<p>I guess I should make the call after starting the supervision tree. So I’ll rewrite the end of the <code class="highlighter-rouge">start</code> function like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">with</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">pid</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="n">opts</span><span class="p">),</span>
<span class="ss">:ok</span> <span class="o"><-</span> <span class="no">GenEvent</span><span class="o">.</span><span class="n">add_handler</span><span class="p">(</span><span class="ss">:toy_event_manager</span><span class="p">,</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandler</span><span class="p">,</span> <span class="no">nil</span><span class="p">),</span>
<span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">pid</span><span class="p">}</span></code></pre></figure>
<p>If we start the tree successfully then add the handler. This seems to work or at least all the exisitng tests pass which means the application came up. But, this code is a little ugly. Why does the application module need to know <code class="highlighter-rouge">PlayChannel.Toy.UpdateEventHandler</code>’s’ initial state? Let’s encapsulate this a bit better. I’m adding the function below to <code class="highlighter-rouge">PlayChannel.Toy.UpdateEventHandler</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">register_with_manager</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span> <span class="k">do</span>
<span class="no">GenEvent</span><span class="o">.</span><span class="n">add_handler</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="bp">__MODULE__</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>Then I can call this from the application module instead:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">with</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">pid</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="n">opts</span><span class="p">),</span>
<span class="ss">:ok</span> <span class="o"><-</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">UpdateEventHandler</span><span class="o">.</span><span class="n">register_with_manager</span><span class="p">(</span><span class="ss">:toy_event_manager</span><span class="p">),</span>
<span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">pid</span><span class="p">}</span></code></pre></figure>
<p>That’s a little better. Though the line is still a bit too long. But this has better separation of concerns. <code class="highlighter-rouge">UpdateEventHandler</code> knows its initial state. And <code class="highlighter-rouge">PlayChannel</code> knows the name of the GenServer process. Nice and clean.</p>
<p>Nice and clean is well enough, but does this actually work?</p>
<h2 id="sending-the-event">Sending the Event</h2>
<p>The last thing we need to do is to modify the controller to send an event instead of calling the channel directly like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="err">@@</span> <span class="o">-</span><span class="m">45</span><span class="p">,</span><span class="m">7</span> <span class="o">+</span><span class="m">45</span><span class="p">,</span><span class="m">7</span> <span class="err">@@</span> <span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">ToyController</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">Repo</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">changeset</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">toy</span><span class="p">}</span> <span class="o">-></span>
<span class="o">-</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">ToyChannel</span><span class="o">.</span><span class="n">broadcast_change</span><span class="p">(</span><span class="n">toy</span><span class="p">)</span>
<span class="o">+</span> <span class="no">GenEvent</span><span class="o">.</span><span class="n">notify</span><span class="p">(</span><span class="ss">:toy_event_manager</span><span class="p">,</span> <span class="p">{</span><span class="ss">:update</span><span class="p">,</span> <span class="n">toy</span><span class="p">})</span>
<span class="n">conn</span>
<span class="o">|></span> <span class="n">put_flash</span><span class="p">(</span><span class="ss">:info</span><span class="p">,</span> <span class="sd">"</span><span class="s2">Toy updated successfully."</span><span class="p">)</span></code></pre></figure>
<p>And this works. The show page still updates in realtime in response to edits.</p>
<h2 id="refactor">Refactor</h2>
<p>Now that things are working I want to refactor a little. We made some nice cleanups in the last section between the <code class="highlighter-rouge">PlayChannel</code> application module and the <code class="highlighter-rouge">PlayChannel.Toy.UpdateEventHandler</code>. But we’ve taken a step backwards in this regard with these changes to the controller.</p>
<p>The issue comes down to where to put the knowledge about the name of the GenEvent process and other details. I think I want them in a new module:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">EventManager</span> <span class="k">do</span>
<span class="k">end</span></code></pre></figure>
<p>But what goes in here? There are few things we’ve written, so far, related to the event manager:</p>
<ul>
<li>worker specification</li>
<li>notify</li>
<li>registration - generally, not specifically to any particular handler.</li>
</ul>
<p>I think this should do it:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">EventManager</span> <span class="k">do</span>
<span class="nv">@name</span> <span class="ss">:toy_event_manager</span>
<span class="k">def</span> <span class="n">child_spec</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="no">Supervisor</span><span class="o">.</span><span class="no">Spec</span><span class="o">.</span><span class="n">worker</span><span class="p">(</span><span class="no">GenEvent</span><span class="p">,</span> <span class="p">[[</span><span class="ss">name:</span> <span class="nv">@name</span><span class="p">]])</span>
<span class="nv">@doc</span> <span class="sd">"""
Notifies the event manager with an event of the form {:update, toy}
"""</span>
<span class="k">def</span> <span class="n">update</span><span class="p">(</span><span class="n">toy</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">GenEvent</span><span class="o">.</span><span class="n">notify</span><span class="p">(</span><span class="nv">@name</span><span class="p">,</span> <span class="p">{</span><span class="ss">:update</span><span class="p">,</span> <span class="n">toy</span><span class="p">})</span>
<span class="k">def</span> <span class="n">register</span><span class="p">(</span><span class="n">handler</span><span class="p">,</span> <span class="n">args</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">GenEvent</span><span class="o">.</span><span class="n">add_handler</span><span class="p">(</span><span class="nv">@name</span><span class="p">,</span> <span class="n">handler</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>Here I store the name for the process in the module attribute <code class="highlighter-rouge">@name</code>. An alternative might just be to use <code class="highlighter-rouge">__MODULE__</code> as the process name.</p>
<p>Next, we have the <code class="highlighter-rouge">child_spec</code> function. This can be used when building the supervision tree like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">children</span> <span class="o">=</span> <span class="p">[</span>
<span class="c1"># Start the endpoint when the application starts</span>
<span class="n">supervisor</span><span class="p">(</span><span class="no">PlayChannel</span><span class="o">.</span><span class="no">Endpoint</span><span class="p">,</span> <span class="p">[]),</span>
<span class="c1"># Start the Ecto repository</span>
<span class="n">supervisor</span><span class="p">(</span><span class="no">PlayChannel</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span> <span class="p">[]),</span>
<span class="c1"># Here you could define other workers and supervisors as children</span>
<span class="c1"># worker(PlayChannel.Worker, [arg1, arg2, arg3]),</span>
<span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">EventManager</span><span class="o">.</span><span class="n">child_spec</span>
<span class="p">]</span></code></pre></figure>
<p>Next, in <code class="highlighter-rouge">PlayChannel.Toy.EventManager</code> we have the <code class="highlighter-rouge">update/1</code> function which notifies with the <code class="highlighter-rouge">:update</code> event. I’ve choosen to document this function because the format of the event needs to be used outside this module. That is, the handlers need to be able to match on this event format. We can call <code class="highlighter-rouge">update/1</code> from the controller like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="err">@@</span> <span class="o">-</span><span class="m">45</span><span class="p">,</span><span class="m">7</span> <span class="o">+</span><span class="m">45</span><span class="p">,</span><span class="m">7</span> <span class="err">@@</span> <span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">ToyController</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">Repo</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">changeset</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">toy</span><span class="p">}</span> <span class="o">-></span>
<span class="o">-</span> <span class="no">GenEvent</span><span class="o">.</span><span class="n">notify</span><span class="p">(</span><span class="ss">:toy_event_manager</span><span class="p">,</span> <span class="p">{</span><span class="ss">:update</span><span class="p">,</span> <span class="n">toy</span><span class="p">})</span>
<span class="o">+</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">EventManager</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">toy</span><span class="p">)</span>
<span class="n">conn</span>
<span class="o">|></span> <span class="n">put_flash</span><span class="p">(</span><span class="ss">:info</span><span class="p">,</span> <span class="sd">"</span><span class="s2">Toy updated successfully."</span><span class="p">)</span></code></pre></figure>
<p>Finally, in <code class="highlighter-rouge">PlayChannel.Toy.EventManager</code> we have the <code class="highlighter-rouge">register/2</code> method. We’ll call this from
<code class="highlighter-rouge">PlayChannel.Toy.UpdateEventHandler.register_with_manager</code>. With this change <code class="highlighter-rouge">register_with_manager</code> no longer needs to take a pid or name as an argument. So the function looks like this now:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">register_with_manager</span> <span class="k">do</span>
<span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="o">.</span><span class="no">EventManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>You can probably guess what the change to the application to call <code class="highlighter-rouge">register_with_manager</code> now.</p>
<p>And with these changes I can still edit a toy in one window and see it update live in another!</p>
<h2 id="conclusion">Conclusion</h2>
<p>I’m pretty happy with this now. We have some nice clean interfaces for our event manager in the <code class="highlighter-rouge">PlayChannel.Toy.EventManager</code> module. We have a clean handler. There’s certinaly more that could be cleaned up here. For example, it would nice to add some aliases in the various modules so I could shorten up these module names. But I’ll save that for later.</p>
<p>We’ve also spent a lot of work to insert this abstract event manager and event handler into something that was already working. One might ask why bother? Well, for the simple example we have this may not be a big deal. But suppose our application grows in complexity, then we could end up filling our controller with a long list of modules that need to be updated every time a module changes. Is this really something we want our controller doing? I think not, we want to keep our controller small. This mechanism also reduces coupling. The controller only needs to know about the event manager. It doesn’t need to know anything about the handlers or the channels they represent.</p>
<p><a href="http://learningelixir.joekain.com/using-genevent-to-notify-a-channel/">Read more...</a></p>
http://learningelixir.joekain.com/pushing-model-changes-to-a-phoenix-channel2016-02-18T00:00:00-08:002016-02-18T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>I’ve been writing a few posts about Ecto lately and in this post I want to talk about how to publish changes to an Ecto model to a Phoenix channel. This will be my first post on Phoenix.</p>
<p>By the way, I know that “Ecto model” is a deprecated term, but what is the right term to use?</p>
<h2 id="motivation">Motivation</h2>
<p>In a private project that I’ve been working on I wanted to publish model updates to a channel so that the front end could receive them and make updates to the user interface. I started by reading the <a href="http://www.phoenixframework.org/docs/channels">Phoenix channel guide</a> which was quite helpful for getting setup. But one thing that I was missing was how to write the model updates to the channel from my controller.</p>
<p>Most of what I’ve read about channels have data that originates within the channel. For example, a client writes data to the topic and the Channel broadcasts it to all other subscribers. This stays completely within the channel code.</p>
<p>But, my use case was different. I have RESTful interfaces that update my model and based on that update I want to broadcast the state of the model to the channel topic subscribers. Arguably, I could have changed my interfaces to send updates on the channel but I didn’t want to :)</p>
<h2 id="the-phoenix-endpoint-api">The Phoenix Endpoint API</h2>
<p>I spent some time digging and eventually found what I needed in order to do what I wanted. The interface is in the <code class="highlighter-rouge">Endpoint</code>:</p>
<blockquote>
<p>broadcast(topic, event, msg) - broadcasts a msg with as event in the given topic.</p>
</blockquote>
<p>The <code class="highlighter-rouge">Endpoint</code> provides a very useful API. I highly recommend reading through the <a href="http://hexdocs.pm/phoenix/Phoenix.html">Phoenix framework docs</a>. There is so much useful information in there.</p>
<h2 id="an-example">An Example</h2>
<p>Let’s build up an example to see how to use <code class="highlighter-rouge">Endpoint.broadcast/3</code>. There’s a lot to do before we get to the point where we can use <code class="highlighter-rouge">Endpoint.broadcast/3</code> so let’s dive in and write some code. First we’ll setup a basic Phoenix application following the <a href="http://www.phoenixframework.org/docs/up-and-running">Up And Running Guide</a>:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix phoenix.new play_channel
$ cd play_channel
$ mix ecto.create
</code></pre>
</div>
<p>I answered <code class="highlighter-rouge">Y</code> when it asked if I wanted to fetch and install the dependencies.</p>
<p>I’ve pushed the example code to <a href="https://github.com/joekain/play_channel">a github repo</a> if you want to follow along.</p>
<h3 id="building-a-model">Building a Model</h3>
<p>Let’s generate a model with a page that can be used to update it.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix phoenix.gen.html Toy toys name:string color:string age:integer
* creating web/controllers/toy_controller.ex
* creating web/templates/toy/edit.html.eex
* creating web/templates/toy/form.html.eex
* creating web/templates/toy/index.html.eex
* creating web/templates/toy/new.html.eex
* creating web/templates/toy/show.html.eex
* creating web/views/toy_view.ex
* creating test/controllers/toy_controller_test.exs
* creating priv/repo/migrations/20160217224802_create_toy.exs
* creating web/models/toy.ex
* creating test/models/toy_test.exs
Add the resource to your browser scope in web/router.ex:
resources "/toys", ToyController
Remember to update your repository by running migrations:
$ mix ecto.migrate
</code></pre>
</div>
<p>I’ll follow the instructions and add the resource to the router and migrate the db.</p>
<p>Next I’ll fire up the server with <code class="highlighter-rouge">mix phoenix.server</code> and then navigate to http://localhost:4000/toys. On this page I can add toys. To follow along with the rest of this post make sure you add at least one toy.</p>
<h3 id="publishing-a-model-on-a-channel">Publishing a Model On a Channel</h3>
<p>Now I want to create a channel that broadcasts changes to a given toy. We’ll start by generating a channel for toys using Phoenix’s generator:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix phoenix.gen.channel Toy toys
* creating web/channels/toy_channel.ex
* creating test/channels/toy_channel_test.exs
Add the channel to your `web/channels/user_socket.ex` handler, for example:
channel "toys:lobby", PlayChannel.ToyChannel
</code></pre>
</div>
<p>Again, we follow the instructions and update the user socket:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gu">@@ -4,6 +4,8 @@ defmodule PlayChannel.UserSocket do
</span> ## Channels
# channel "rooms:*", PlayChannel.RoomChannel
<span class="gi">+ channel "toys:*", PlayChannel.ToyChannel
+
</span> ## Transports
transport :websocket, Phoenix.Transports.WebSocket
<span class="err"> # transport :longpoll, Phoenix.Transports.LongPoll</span></code></pre></figure>
<p>Next, let’s take a look at the channel code that was generated:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">ToyChannel</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Web</span><span class="p">,</span> <span class="ss">:channel</span>
<span class="k">def</span> <span class="n">join</span><span class="p">(</span><span class="sd">"</span><span class="s2">toys:lobby"</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span> <span class="k">do</span>
<span class="k">if</span> <span class="n">authorized?</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">socket</span><span class="p">}</span>
<span class="k">else</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="p">%{</span><span class="ss">reason:</span> <span class="sd">"</span><span class="s2">unauthorized"</span><span class="p">}}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Channels can be used in a request/response fashion</span>
<span class="c1"># by sending replies to requests from the client</span>
<span class="k">def</span> <span class="n">handle_in</span><span class="p">(</span><span class="sd">"</span><span class="s2">ping"</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">payload</span><span class="p">},</span> <span class="n">socket</span><span class="p">}</span>
<span class="k">end</span>
<span class="c1"># It is also common to receive messages from the client and</span>
<span class="c1"># broadcast to everyone in the current topic (toys:lobby).</span>
<span class="k">def</span> <span class="n">handle_in</span><span class="p">(</span><span class="sd">"</span><span class="s2">shout"</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span> <span class="k">do</span>
<span class="n">broadcast</span> <span class="n">socket</span><span class="p">,</span> <span class="sd">"</span><span class="s2">shout"</span><span class="p">,</span> <span class="n">payload</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">socket</span><span class="p">}</span>
<span class="k">end</span>
<span class="c1"># This is invoked every time a notification is being broadcast</span>
<span class="c1"># to the client. The default implementation is just to push it</span>
<span class="c1"># downstream but one could filter or change the event.</span>
<span class="k">def</span> <span class="n">handle_out</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span> <span class="k">do</span>
<span class="n">push</span> <span class="n">socket</span><span class="p">,</span> <span class="n">event</span><span class="p">,</span> <span class="n">payload</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">socket</span><span class="p">}</span>
<span class="k">end</span>
<span class="c1"># Add authorization logic here as required.</span>
<span class="k">defp</span> <span class="n">authorized?</span><span class="p">(</span><span class="n">_payload</span><span class="p">)</span> <span class="k">do</span>
<span class="no">true</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Some notes:</p>
<ul>
<li>We have a <code class="highlighter-rouge">join</code> function though it doesn’t handle the rooms we want. I’ll strip it down to the bare minimum and we’ll come back to it later.</li>
<li>We have <code class="highlighter-rouge">handle_in</code> to receive “ping” and “shout”. At this point I don’t really need any input so I’ll remove these functions.</li>
<li>We have <code class="highlighter-rouge">handle_out</code> to modify broadcast events. I’ll leave this as is.</li>
<li>We have <code class="highlighter-rouge">authorized?</code> used by <code class="highlighter-rouge">join</code>. I’m going to forego authorization for now and will remove this function.</li>
</ul>
<p>We’re left with this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">ToyChannel</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Web</span><span class="p">,</span> <span class="ss">:channel</span>
<span class="k">def</span> <span class="n">join</span><span class="p">(</span><span class="sd">"</span><span class="s2">toys:lobby"</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">socket</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_out</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span> <span class="k">do</span>
<span class="n">push</span> <span class="n">socket</span><span class="p">,</span> <span class="n">event</span><span class="p">,</span> <span class="n">payload</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">socket</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Now for the join function, I want one topic per toy. So the pattern match will look like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">join</span><span class="p">(</span><span class="sd">"</span><span class="s2">toys:"</span> <span class="o"><></span> <span class="n">toy_id</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span></code></pre></figure>
<p>This matches any topic that starts with “toys:” followed by another string. We interpret that second string as the model id.</p>
<p>Let’s also send back a message on join so that we can make sure things are working.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">join</span><span class="p">(</span><span class="sd">"</span><span class="s2">toys:"</span> <span class="o"><></span> <span class="n">toy_id</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="sd">"</span><span class="s2">Joined toys:</span><span class="si">#{</span><span class="n">toy_id</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="n">socket</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<h3 id="connecting-the-front-end-to-the-phoenix-channel">Connecting the Front End to the Phoenix Channel</h3>
<p>It’s time to start working on the front end. I’ll admit that I’m not expert here so if you have any suggestions on how to improve the Javascript presented here I’d be happy to hear them.</p>
<p>We’ll use Javascript to update the show page. Currently, the page template looks like this:</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><h2></span>Show toy<span class="nt"></h2></span>
<span class="nt"><ul></span>
<span class="nt"><li></span>
<span class="nt"><strong></span>Name:<span class="nt"></strong></span>
<span class="err"><</span>%= @toy.name %>
<span class="nt"></li></span>
<span class="nt"><li></span>
<span class="nt"><strong></span>Color:<span class="nt"></strong></span>
<span class="err"><</span>%= @toy.color %>
<span class="nt"></li></span>
<span class="nt"><li></span>
<span class="nt"><strong></span>Age:<span class="nt"></strong></span>
<span class="err"><</span>%= @toy.age %>
<span class="nt"></li></span>
<span class="nt"></ul></span>
<span class="err"><</span>%= link "Edit", to: toy_path(@conn, :edit, @toy) %>
<span class="err"><</span>%= link "Back", to: toy_path(@conn, :index) %></code></pre></figure>
<p>We’ll want to generate the content of <code class="highlighter-rouge"><ul></code> dynamically. Let’s remove the <code class="highlighter-rouge"><li></code> elements and add an id so we can reference the <code class="highlighter-rouge"><ul></code>. We’ll also add a data item to communicate in the relevant toy id to our Javascript:</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><h2></span>Show toy<span class="nt"></h2></span>
<span class="nt"><ul</span> <span class="na">id=</span><span class="s">"show-list"</span> <span class="na">data-id=</span><span class="s"><%=</span> <span class="err">@</span><span class="na">toy</span><span class="err">.</span><span class="na">id</span> <span class="err">%</span><span class="nt">></span>>
<span class="nt"></ul></span>
<span class="err"><</span>%= link "Edit", to: toy_path(@conn, :edit, @toy) %>
<span class="err"><</span>%= link "Back", to: toy_path(@conn, :index) %></code></pre></figure>
<p>Before we go any further, let’s add jquery to the project using bower. First we’ll need an initial bower.json:</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"play_channel"</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>Then we can install jquery:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ bower install jquery --save
bower cached git://github.com/jquery/jquery-dist.git#2.2.0
bower validate 2.2.0 against git://github.com/jquery/jquery-dist.git#*
bower install jquery#2.2.0
jquery#2.2.0 bower_components/jquery
</code></pre>
</div>
<p>And that’s it. Now brunch will compile jquery into our app.js and make it available for us.</p>
<p>Next, let’s create a new file: web/static/js/toy.js. It needs to join the channel:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kr">import</span> <span class="nx">socket</span> <span class="nx">from</span> <span class="s2">"./socket"</span>
<span class="nx">$</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">ul</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="s2">"ul#show-list"</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">ul</span><span class="p">.</span><span class="nx">length</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">id</span> <span class="o">=</span> <span class="nx">ul</span><span class="p">.</span><span class="nx">data</span><span class="p">(</span><span class="s2">"id"</span><span class="p">)</span>
<span class="kd">var</span> <span class="nx">topic</span> <span class="o">=</span> <span class="s2">"toys:"</span> <span class="o">+</span> <span class="nx">id</span>
<span class="c1">// Join the topic</span>
<span class="kd">let</span> <span class="nx">channel</span> <span class="o">=</span> <span class="nx">socket</span><span class="p">.</span><span class="nx">channel</span><span class="p">(</span><span class="nx">topic</span><span class="p">,</span> <span class="p">{})</span>
<span class="nx">channel</span><span class="p">.</span><span class="nx">join</span><span class="p">()</span>
<span class="p">.</span><span class="nx">receive</span><span class="p">(</span><span class="s2">"ok"</span><span class="p">,</span> <span class="nx">data</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"Joined topic"</span><span class="p">,</span> <span class="nx">topic</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">receive</span><span class="p">(</span><span class="s2">"error"</span><span class="p">,</span> <span class="nx">resp</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"Unable to join topic"</span><span class="p">,</span> <span class="nx">topic</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">}</span>
<span class="p">});</span></code></pre></figure>
<p>This code first imports the socket library that Phoenix generated for us. Then it has a document ready function that builds the topic name from our toy id and then joins the topic. If successful, it logs “Joined topic”.</p>
<p>Before we can make use of toy.js we need to import it into our app.js by adding a line like this:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kr">import</span> <span class="nx">toy</span> <span class="nx">from</span> <span class="s2">"./toy"</span></code></pre></figure>
<p>This works! When I navigate to http://localhost:4000/toys/1 I see</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[Log] Joined topic – "toys:1" (app.js, line 2878)
</code></pre>
</div>
<p>on my Javascript console.</p>
<h3 id="sending-model-data-over-the-channel">Sending Model Data Over the Channel</h3>
<p>Let’s add a new function to our channel module. This function will provide an interface to other parts of our application to allow it to broadcast a Toy model over the channel.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">broadcast_change</span><span class="p">(</span><span class="n">toy</span><span class="p">)</span> <span class="k">do</span>
<span class="n">payload</span> <span class="o">=</span> <span class="p">%{</span>
<span class="sd">"</span><span class="s2">name"</span> <span class="o">=></span> <span class="n">toy</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
<span class="sd">"</span><span class="s2">color"</span> <span class="o">=></span> <span class="n">toy</span><span class="o">.</span><span class="n">color</span><span class="p">,</span>
<span class="sd">"</span><span class="s2">age"</span> <span class="o">=></span> <span class="n">toy</span><span class="o">.</span><span class="n">age</span><span class="p">,</span>
<span class="sd">"</span><span class="s2">id"</span> <span class="o">=></span> <span class="n">toy</span><span class="o">.</span><span class="n">id</span>
<span class="p">}</span>
<span class="no">PlayChannel</span><span class="o">.</span><span class="no">Endpoint</span><span class="o">.</span><span class="n">broadcast</span><span class="p">(</span><span class="sd">"</span><span class="s2">toys:</span><span class="si">#{</span><span class="n">toy</span><span class="o">.</span><span class="n">id</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="sd">"</span><span class="s2">change"</span><span class="p">,</span> <span class="n">payload</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p><code class="highlighter-rouge">broadcast_change</code> does two things. First, it creates the payload which is map representing our
toy. This map will be serialized to JSON before being broadcast. Second, we use the <code class="highlighter-rouge">Endpoint.broadcast</code> function to broadcast the map over the channel.</p>
<p>The first argument to <code class="highlighter-rouge">PlayChannel.Endpoint.broadcast</code> is the topic name. We use our topic format of “toys:” followed by the model id. We encapsulate this knowledge inside this function in our channel code. The controller that will end up calling <code class="highlighter-rouge">broadcast_change</code> doesn’t need to know the topic name. It just needs a Toy model.</p>
<p>Also, note that we broadcast an event called “change” which caries the payload. We’ll need to respond to this “change” event in the front end.</p>
<p>Now that we have this function we can call it from our controller when a toy is updated:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"> def update(conn, %{"id" => id, "toy" => toy_params}) do
toy = Repo.get!(Toy, id)
changeset = Toy.changeset(toy, toy_params)
case Repo.update(changeset) do
{:ok, toy} ->
<span class="gi">+ PlayChannel.ToyChannel.broadcast_change(toy)
+
</span> conn
|> put_flash(:info, "Toy updated successfully.")
<span class="err"> |> redirect(to: toy_path(conn, :show, toy))</span></code></pre></figure>
<h3 id="rendering-the-model-on-the-show-page">Rendering the Model on the Show Page</h3>
<p>The final step is to go back to the front end Javascript and react to the change event.</p>
<p>We’ll start by logging the event by adding this code to our document ready function after joining the channel.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">channel</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"change"</span><span class="p">,</span> <span class="nx">toy</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"Change:"</span><span class="p">,</span> <span class="nx">toy</span><span class="p">);</span>
<span class="p">})</span></code></pre></figure>
<p>Now, in order to try this out I need two browser windows open.</p>
<p>In the first window I navigate to the show page for my toy at <a href="http://localhost:4000/toys/1">http://localhost:4000/toys/1</a>. Then I open the Javascript console.</p>
<p>In the second browser window I navigate to the edit page for my toy at: <a href="http://localhost:4000/toys/1/edit">http://localhost:4000/toys/1/edit</a>. I modify the color and press “Submit”.</p>
<p>Back in the first browser window I see the following log entry:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[Log] Change: – {name: "Ball", id: 1, color: "Green", …} (app.js, line 2899)
</code></pre>
</div>
<p>Our channel is sending us data!</p>
<p>Now, let’s write up some Javascript to generate the show page contents we want. Remember we left ourselves with an empty <code class="highlighter-rouge"><ul></code> list.</p>
<p>We’ll encapsulate our Javascript into a class:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kr">export</span> <span class="kd">var</span> <span class="nx">Toy</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">show</span><span class="p">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">ul</span><span class="p">,</span> <span class="nx">toy</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">ul</span><span class="p">.</span><span class="nx">empty</span><span class="p">();</span>
<span class="nx">ul</span>
<span class="p">.</span><span class="nx">append</span><span class="p">(</span><span class="s1">'<li><strong>Name:</strong> '</span> <span class="o">+</span> <span class="nx">toy</span><span class="p">.</span><span class="nx">name</span> <span class="o">+</span> <span class="s1">'</li>'</span><span class="p">)</span>
<span class="p">.</span><span class="nx">append</span><span class="p">(</span><span class="s1">'<li><strong>Color:</strong> '</span> <span class="o">+</span> <span class="nx">toy</span><span class="p">.</span><span class="nx">color</span> <span class="o">+</span> <span class="s1">'</li>'</span><span class="p">)</span>
<span class="p">.</span><span class="nx">append</span><span class="p">(</span><span class="s1">'<li><strong>Age:</strong> '</span> <span class="o">+</span> <span class="nx">toy</span><span class="p">.</span><span class="nx">age</span> <span class="o">+</span> <span class="s1">'</li>'</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>Our new function, <code class="highlighter-rouge">Toy.show</code>, takes two arguments. The first is the <code class="highlighter-rouge"><ul></code> element to update and the second is an object with our Toy model data. The function cleans out the <code class="highlighter-rouge"><ul></code> and then fills it with our desired content.</p>
<p>We can call <code class="highlighter-rouge">Toy.show</code> from our channel <code class="highlighter-rouge">on</code> function like this:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">channel</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"change"</span><span class="p">,</span> <span class="nx">toy</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"Change:"</span><span class="p">,</span> <span class="nx">toy</span><span class="p">);</span>
<span class="nx">Toy</span><span class="p">.</span><span class="nx">show</span><span class="p">(</span><span class="nx">ul</span><span class="p">,</span> <span class="nx">toy</span><span class="p">);</span>
<span class="p">});</span></code></pre></figure>
<p>With this our show page fills with content after we update the Toy.</p>
<h3 id="sending-initial-data-over-the-channel">Sending Initial Data Over the Channel</h3>
<p>We have one remaining issue, nothing is shown on the show page until an update occurs. We want the page to show data when it first visited. We can fix this by sending the model data on join like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">join</span><span class="p">(</span><span class="sd">"</span><span class="s2">toys:"</span> <span class="o"><></span> <span class="n">toy_id</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">PlayChannel</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="no">PlayChannel</span><span class="o">.</span><span class="no">Toy</span><span class="p">,</span> <span class="n">toy_id</span><span class="p">)</span> <span class="k">do</span>
<span class="no">nil</span> <span class="o">-></span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="p">%{</span><span class="ss">reason:</span> <span class="sd">"</span><span class="s2">channel: No such toy </span><span class="si">#{</span><span class="n">toy_id</span><span class="si">}</span><span class="s2">"</span><span class="p">}}</span>
<span class="n">toy</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">toy_to_map</span><span class="p">(</span><span class="n">toy</span><span class="p">),</span> <span class="n">socket</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">toy_to_map</span><span class="p">(</span><span class="n">toy</span><span class="p">)</span> <span class="k">do</span>
<span class="p">%{</span>
<span class="sd">"</span><span class="s2">name"</span> <span class="o">=></span> <span class="n">toy</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
<span class="sd">"</span><span class="s2">color"</span> <span class="o">=></span> <span class="n">toy</span><span class="o">.</span><span class="n">color</span><span class="p">,</span>
<span class="sd">"</span><span class="s2">age"</span> <span class="o">=></span> <span class="n">toy</span><span class="o">.</span><span class="n">age</span><span class="p">,</span>
<span class="sd">"</span><span class="s2">id"</span> <span class="o">=></span> <span class="n">toy</span><span class="o">.</span><span class="n">id</span>
<span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>I’ve extracted <code class="highlighter-rouge">toy_to_map</code> from <code class="highlighter-rouge">broadcast_change</code>.</p>
<p>Note, this is not a change event, but it available in the “ok” handler for join. So we can add a call to <code class="highlighter-rouge">Toy.show</code> like this:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">channel</span><span class="p">.</span><span class="nx">join</span><span class="p">()</span>
<span class="p">.</span><span class="nx">receive</span><span class="p">(</span><span class="s2">"ok"</span><span class="p">,</span> <span class="nx">toy</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"Joined topic"</span><span class="p">,</span> <span class="nx">topic</span><span class="p">);</span>
<span class="nx">Toy</span><span class="p">.</span><span class="nx">show</span><span class="p">(</span><span class="nx">ul</span><span class="p">,</span> <span class="nx">toy</span><span class="p">);</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">receive</span><span class="p">(</span><span class="s2">"error"</span><span class="p">,</span> <span class="nx">resp</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"Unable to join topic"</span><span class="p">,</span> <span class="nx">topic</span><span class="p">)</span>
<span class="p">});</span></code></pre></figure>
<p>And with this the model state is shown when we first enter the page and it is updated live whenever and edit occurs.</p>
<h2 id="next-steps">Next Steps</h2>
<p>In this post we showed how to publish models and model updates over a Phoenix channel. This allows the client to update its page in near realtime in response to changes in the server. In building this we learned about the <code class="highlighter-rouge">Endpoint.broadcast</code> function.</p>
<p>Next week I want to continue this project by exploring how to decouple the controller and the channel. Perhaps I can use GenEvent to notify the channel of changes rather than having the controller call the channel directly.</p>
<p><a href="http://learningelixir.joekain.com/pushing-model-changes-to-a-phoenix-channel/">Read more...</a></p>
http://learningelixir.joekain.com/custom-types-in-ecto2016-02-09T00:00:00-08:002016-02-09T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>Last week I wrote a <a href="/fragments-in-ecto/">small example of using fragment to query overlapping date ranges from an Ecto repo</a>. In this post I’ll continue using the same example but will show how I use custom Ecto types to manage data conversion.</p>
<p>I’ll be using Elixir 1.2.1 and the following dependencies (according to mix.lock):</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="p">%{</span><span class="sd">"</span><span class="s2">connection"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:connection</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.0.2"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">db_connection"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:db_connection</span><span class="p">,</span> <span class="sd">"</span><span class="s2">0.2.3"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">decimal"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:decimal</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.1.1"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">ecto"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:ecto</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.1.3"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">poolboy"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:poolboy</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.5.1"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">postgrex"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:postgrex</span><span class="p">,</span> <span class="sd">"</span><span class="s2">0.11.0"</span><span class="p">}}</span></code></pre></figure>
<h2 id="date-formats">Date Formats</h2>
<p>I’m working on a Phoenix project and the frontend generates dates in the format “DD/MM/YYYY”. This is a common format in my locale. However, my model uses the <code class="highlighter-rouge">Ecto.Date</code> type and and won’t cast this format. Consulting the <a href="https://hexdocs.pm/ecto/Ecto.Date.html">documentation</a>, I see that the <code class="highlighter-rouge">cast/1</code> function supports:</p>
<ul>
<li>a binary in the “YYYY-MM-DD” format</li>
<li>a binary in the “YYYY-MM-DD HH:MM:DD” format (may be separated by T and/or followed by “Z”, as in 2014-04-17T14:00:00Z)</li>
<li>a binary in the “YYYY-MM-DD HH:MM:DD.USEC” format (may be separated by T and/or followed by “Z”, as in 2014-04-17T14:00:00.030Z)</li>
<li>a map with “year”, “month” and “day” keys with integer or binaries as values</li>
<li>a map with :year, :month and :day keys with integer or binaries as values</li>
<li>a tuple with {year, month, day} as integers or binaries</li>
<li>an Ecto.Date struct itself</li>
</ul>
<p>Currently, I have some conversion code to translate the date in “DD/MM/YYYY” format to one of the formats accepted by <code class="highlighter-rouge">Ecto.Date.cast/1</code> but I have the code in my controller. This is the wrong place for such code. I want to build something cleaner.</p>
<h2 id="example-code">Example Code</h2>
<p>I’ll be building the code for this post based on the code from last week’s post. If you haven’t read last weeks post you may want to. Though the custom type we will develop here will be somewhat orthogonal to the rest of the project. One thing I’ll from the old code is the Ecto setup so I won’t repeat that here.</p>
<p>The github repo for the sample code is <a href="https://github.com/joekain/date_ranges">here</a>.</p>
<h2 id="a-custom-ecto-type">A Custom Ecto Type</h2>
<p>Given that my application needs to support a single date format, my thought is to write a custom Ecto type that accepts this format in <code class="highlighter-rouge">cast/1</code>.</p>
<p>For clarity, in this post I’ll call my new type <code class="highlighter-rouge">CustomDate</code> or in full: <code class="highlighter-rouge">DateRanges.CustomDate</code>. As part of a real application I would name it based on its function. But in this post its function is to demonstrate a custom type. There are a few other date types involved in the project so I hope this name makes it clear that <code class="highlighter-rouge">CustomDate</code> is our custom ecto type.</p>
<p>Looking at the <a href="https://hexdocs.pm/ecto/Ecto.Type.html">documentation for <code class="highlighter-rouge">Ecto.Type</code></a>, we see that <code class="highlighter-rouge">Ecto.Type</code> is a behaviour.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Ecto.Type behaviour
Defines functions and the Ecto.Type behaviour for implementing custom types.
A custom type expects 4 functions to be implemented, all documented and described below.
</code></pre>
</div>
<p>The documentation goes on to describe the 4 functions:</p>
<ul>
<li><code class="highlighter-rouge">type</code> should output the name of the db type</li>
<li><code class="highlighter-rouge">cast</code> should receive any type and output your custom Ecto type</li>
<li><code class="highlighter-rouge">load</code> should receive the db type and output your custom Ecto type</li>
<li><code class="highlighter-rouge">dump</code> should receive your custom Ecto type and output the db type</li>
</ul>
<p>Now that we have a basic understanding of what’s expected, let’s start implementing.</p>
<h3 id="ecto-type-cast">Ecto Type Cast</h3>
<p>We’ll start with <code class="highlighter-rouge">cast/1</code> and to get us going we’ll write a test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it can convert the MM/DD/YYYY format to a date"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="no">CustomDate</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="sd">"</span><span class="s2">2/9/2016"</span><span class="p">)</span> <span class="o">==</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="sx">??</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>Hmm, I’m not sure what <code class="highlighter-rouge">cast</code> should return here. I know its an :ok tupple but I don’t know what goes in the second field of the tupple. I want the same thing that Ecto.Date would return so let’s write a quick test to understand its return type:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it can convert the MM/DD/YYYY format to a date"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="sd">"</span><span class="s2">2016-09-02"</span><span class="p">)</span> <span class="o">==</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="m">5</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>The “5” is clearly wrong. I put it in the test to force a failure - this is the failure:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test it can convert the MM/DD/YYYY format to a date (DateRangeTest)
test/custom_date_test.exs:11
Assertion with == failed
code: Ecto.Date.cast("2016-02-09") == {:ok, 5}
lhs: {:ok, #Ecto.Date<2016-02-09>}
rhs: {:ok, 5}
stacktrace:
test/custom_date_test.exs:12
</code></pre>
</div>
<p>Ah, so <code class="highlighter-rouge">Ecto.Date.cast</code> casts to the <code class="highlighter-rouge">Ecto.Date</code> struct. I don’t think I want a struct of my own. Instead I think I will wrap <code class="highlighter-rouge">Ecto.Date</code>. That is, my cast should also return an <code class="highlighter-rouge">Ecto.Date</code>. Now that I’ve figured this out, I can throw away this test. It doesn’t test our code, and we shouldn’t be testing <code class="highlighter-rouge">Ecto.Date</code> functionality. But we’ve learned something from our experiment and can put that new knowledge to work and write our own <code class="highlighter-rouge">CustomDate</code> test that looks like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it can convert the MM/DD/YYYY format to a date"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="no">CustomDate</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="sd">"</span><span class="s2">2/9/2016"</span><span class="p">)</span> <span class="o">==</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="sd">"</span><span class="s2">2016-02-09"</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>And here’s our first real failure:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test it can convert the MM/DD/YYYY format to a date (DateRangeTest)
test/custom_date_test.exs:7
** (UndefinedFunctionError) undefined function CustomDate.cast/1 (module CustomDate is not available)
stacktrace:
CustomDate.cast("2/9/2016")
test/custom_date_test.exs:8
</code></pre>
</div>
<p>I need to start writing my CustomDate module. Let’s begin with a new file called custom_date.ex:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">DateRanges</span><span class="o">.</span><span class="no">CustomDate</span> <span class="k">do</span>
<span class="nv">@behaviour</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Type</span>
<span class="k">def</span> <span class="n">cast</span><span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_binary</span><span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="k">do</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>First I’m declaring that this implements the <code class="highlighter-rouge">Ecto.Type</code> behavior though we haven’t implemented that behaviour yet.</p>
<p>Second, I’ve written an empty <code class="highlighter-rouge">cast/1</code> function for strings. Now the test fails like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test it can convert the MM/DD/YYYY format to a date (DateRangeTest)
test/custom_date_test.exs:7
Assertion with == failed
code: CustomDate.cast("2/9/2016") == Ecto.Date.cast("2016-02-09")
lhs: nil
rhs: {:ok, #Ecto.Date<2016-02-09>}
stacktrace:
test/custom_date_test.exs:8
</code></pre>
</div>
<p>Good, so our function is being called. We just need to return the right value.</p>
<p>This is one possible solution, using a regular expression to parse the date and then converting to a form that <code class="highlighter-rouge">Ecto.Date</code> can parse:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">DateRanges</span><span class="o">.</span><span class="no">CustomDate</span> <span class="k">do</span>
<span class="nv">@behaviour</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Type</span>
<span class="k">def</span> <span class="n">cast</span><span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_binary</span><span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">Regex</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="sr">~r/^([0-9]+)\/([0-9]+)\/([0-9]+)$/</span><span class="p">,</span> <span class="n">string</span><span class="p">)</span> <span class="k">do</span>
<span class="p">[</span><span class="n">_match</span><span class="p">,</span> <span class="n">m</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">y</span><span class="p">]</span> <span class="o">-></span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span> <span class="p">{</span><span class="n">y</span><span class="p">,</span> <span class="n">m</span><span class="p">,</span> <span class="n">d</span><span class="p">}</span>
<span class="no">nil</span> <span class="o">-></span> <span class="ss">:error</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>And with this our test passes.</p>
<p>We need to accept a few more formats. For example, if we already have an <code class="highlighter-rouge">Ecto.Date</code> we should take it as is:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it should accept Ecto.Date"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">ed</span><span class="p">}</span> <span class="o">=</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="sd">"</span><span class="s2">2016-02-09"</span><span class="p">)</span>
<span class="n">assert</span> <span class="n">match?</span><span class="p">({</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">_</span><span class="p">},</span> <span class="no">CustomDate</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="n">ed</span><span class="p">))</span>
<span class="k">end</span></code></pre></figure>
<p>And of course the test fails:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test it should accept Ecto.Date (CustomDateTest)
test/custom_date_test.exs:10
** (FunctionClauseError) no function clause matching in DateRanges.CustomDate.cast/1
stacktrace:
(date_ranges) lib/date_ranges/custom_date.ex:4: DateRanges.CustomDate.cast({:ok, #Ecto.Date<2016-02-09>})
test/custom_date_test.exs:12
</code></pre>
</div>
<p>because we need to implement another clause for our cast function:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">cast</span><span class="p">(%</span><span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="p">{}</span> <span class="o">=</span> <span class="n">date</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="n">date</span><span class="p">)</span></code></pre></figure>
<p>I decided to call <code class="highlighter-rouge">Ecto.Date.cast/1</code> here. It might not be necessary but why second guess <code class="highlighter-rouge">Ecto.Date</code>?</p>
<p>Are there other types we should handle? Well, thinking about it I think I want to support most of the other cases that <code class="highlighter-rouge">Ecto.Date</code> supports. That is:</p>
<ul>
<li>a map with “year”, “month” and “day” keys with integer or binaries as values</li>
<li>a map with :year, :month and :day keys with integer or binaries as values</li>
<li>a tuple with {year, month, day} as integers or binaries</li>
</ul>
<p>Again, these can be passed through to <code class="highlighter-rouge">Ecto.Date.cast/1</code>. Let’s add one more test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it should accept date tuples"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="no">CustomDate</span><span class="o">.</span><span class="n">cast</span><span class="p">({</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">9</span><span class="p">})</span> <span class="o">==</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="sd">"</span><span class="s2">2016-02-09"</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>the test fails because we need another clause in our <code class="highlighter-rouge">cast</code> function. This implementation will work:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">cast</span><span class="p">({</span><span class="n">y</span><span class="p">,</span> <span class="n">m</span><span class="p">,</span> <span class="n">d</span><span class="p">}</span> <span class="o">=</span> <span class="n">date</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="n">date</span><span class="p">)</span></code></pre></figure>
<p>The tests passes. But notice that this is almost the same function we wrote to accept an <code class="highlighter-rouge">Ecto.Date</code> structure. The only difference is in the accepted pattern. We can combine the two clauses into:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">cast</span><span class="p">(</span><span class="n">date</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="n">date</span><span class="p">)</span></code></pre></figure>
<p>The whole module looks like this now:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">DateRanges</span><span class="o">.</span><span class="no">CustomDate</span> <span class="k">do</span>
<span class="nv">@behaviour</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Type</span>
<span class="k">def</span> <span class="n">cast</span><span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_binary</span><span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">Regex</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="sr">~r/^([0-9]+)\/([0-9]+)\/([0-9]+)$/</span><span class="p">,</span> <span class="n">string</span><span class="p">)</span> <span class="k">do</span>
<span class="p">[</span><span class="n">_match</span><span class="p">,</span> <span class="n">m</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">y</span><span class="p">]</span> <span class="o">-></span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span> <span class="p">{</span><span class="n">y</span><span class="p">,</span> <span class="n">m</span><span class="p">,</span> <span class="n">d</span><span class="p">}</span>
<span class="no">nil</span> <span class="o">-></span> <span class="ss">:error</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">cast</span><span class="p">(</span><span class="n">date</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="n">date</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>With this change the tests still pass. However, we have expanded our functionality and all of the types accepted by <code class="highlighter-rouge">Ecto.Date.cast/1</code> should work now. But, our <code class="highlighter-rouge">cast(string)</code> comes first and will override <code class="highlighter-rouge">Ecto.Date</code>’s version.</p>
<p>I could write tests for all the types <code class="highlighter-rouge">Ecto.Date</code> supports but, I won’t. I’m leaving this responsibility for these other formats to Ecto and its tests. One nice thing about wrapping <code class="highlighter-rouge">Ecto.Date</code> this way is that Ecto could add support for new formats and <code class="highlighter-rouge">CustomDate</code> would inherit them automatically.</p>
<h3 id="dumping-dates">Dumping Dates</h3>
<p>The next step in implementing the <code class="highlighter-rouge">Ecto.Type</code> behavior is to write a <code class="highlighter-rouge">dump</code> funciton. This function should take our internal type and generate a value appropriate for the database. Since we are wrapping <code class="highlighter-rouge">Ecto.Date</code> we should be able to leverage it’s implementation.</p>
<p>But, we start with a test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it should dump dates"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">ed</span><span class="p">}</span> <span class="o">=</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="sd">"</span><span class="s2">2016-02-09"</span><span class="p">)</span>
<span class="n">assert</span> <span class="no">CustomDate</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">ed</span><span class="p">)</span> <span class="o">==</span> <span class="sx">??</span>
<span class="k">end</span></code></pre></figure>
<p>Hmm, again I’m not sure what to expect here. What does <code class="highlighter-rouge">Ecto.Date</code> dump? Again, we’ll write a test to experiment:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it should dump dates"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">ed</span><span class="p">}</span> <span class="o">=</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="sd">"</span><span class="s2">2016-02-09"</span><span class="p">)</span>
<span class="n">assert</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">ed</span><span class="p">)</span> <span class="o">==</span> <span class="m">5</span>
<span class="k">end</span></code></pre></figure>
<p>This test fails and tells us what we should expect:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test it should dump dates (CustomDateTest)
test/custom_date_test.exs:24
Assertion with == failed
code: Ecto.Date.dump(ed) == 5
lhs: {:ok, {2016, 2, 9}}
rhs: 5
stacktrace:
test/custom_date_test.exs:26
</code></pre>
</div>
<p>Now we can write our proper test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it should dump dates"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">ed</span><span class="p">}</span> <span class="o">=</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="sd">"</span><span class="s2">2016-02-09"</span><span class="p">)</span>
<span class="n">assert</span> <span class="no">CustomDate</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">ed</span><span class="p">)</span> <span class="o">==</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">9</span><span class="p">}}</span>
<span class="k">end</span></code></pre></figure>
<p>The test fails and tells us we need to implement <code class="highlighter-rouge">CustomDate.dump</code>. We can do this using delegate like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defdelegate</span> <span class="n">dump</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="ss">to:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span></code></pre></figure>
<p>And with this change the test passes.</p>
<h3 id="loading-dates">Loading Dates</h3>
<p>Now that we have <code class="highlighter-rouge">dump/1</code> implemented we need to implement <code class="highlighter-rouge">load/1</code>. This function should take the database type and return an <code class="highlighter-rouge">Ecto.Date</code>. Again, we will wrap <code class="highlighter-rouge">Ecto.Date</code>’s functionality and we won’t have much to do.</p>
<p>We’ll write a test. Since we’ve just written the <code class="highlighter-rouge">dump/1</code> test we already know what values to expect and won’t need an experimental test to help us out. Our test looks like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it loads dates"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="no">CustomDate</span><span class="o">.</span><span class="n">load</span><span class="p">({</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">9</span><span class="p">})</span> <span class="o">==</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="sd">"</span><span class="s2">2016-02-09"</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>this test fails and tells us to implement the load function. Again, we’ll simply use delegation:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defdelegate</span> <span class="n">load</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="ss">to:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span></code></pre></figure>
<p>And our test passes.</p>
<h3 id="define-the-type">Define the type</h3>
<p>Along the way the Elixir compiler has been giving me warnings. For example:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>lib/date_ranges/custom_date.ex:1: warning: undefined behaviour function dump/1 (for behaviour Ecto.Type)
lib/date_ranges/custom_date.ex:1: warning: undefined behaviour function load/1 (for behaviour Ecto.Type)
</code></pre>
</div>
<p>There is now one warning left:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>lib/date_ranges/custom_date.ex:1: warning: undefined behaviour function type/0 (for behaviour Ecto.Type)
</code></pre>
</div>
<p>We need to define the type we are using. Now that we have <code class="highlighter-rouge">cast/1</code>, <code class="highlighter-rouge">dump/1</code>, and <code class="highlighter-rouge">load/1</code> written we have a pretty good idea of what our type is. The type is <code class="highlighter-rouge">Ecto.Date</code>. So, we can fix this last warning like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">type</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span></code></pre></figure>
<p>And with this we have a complete Ecto type!</p>
<h2 id="testing-our-ectotype">Testing our Ecto.Type</h2>
<p>Now that we have this type we need to put it to use and test it. Here’s a test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it can build a changeset"</span> <span class="k">do</span>
<span class="n">changeset</span> <span class="o">=</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Changeset</span><span class="o">.</span><span class="n">cast</span><span class="p">(%</span><span class="no">DateRange</span><span class="p">{},</span>
<span class="p">%{</span><span class="ss">start:</span> <span class="sd">"</span><span class="s2">2/9/2016"</span><span class="p">,</span> <span class="k">end</span><span class="p">:</span> <span class="sd">"</span><span class="s2">5/9/2016"</span><span class="p">},</span>
<span class="err">~</span><span class="n">w</span><span class="p">(</span><span class="n">start</span> <span class="k">end</span><span class="p">),</span> <span class="err">~</span><span class="n">w</span><span class="p">())</span>
<span class="n">assert</span> <span class="n">changeset</span><span class="o">.</span><span class="n">valid?</span>
<span class="k">end</span></code></pre></figure>
<p>This test fails:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test it can build a changeset (DateRangesTest)
test/date_ranges_test.exs:10
Expected truthy, got false
code: changeset.valid?()
stacktrace:
test/date_ranges_test.exs:14
</code></pre>
</div>
<p>This failure isn’t very enlightening. Let’s change it slightly:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it can build a changeset"</span> <span class="k">do</span>
<span class="n">changeset</span> <span class="o">=</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Changeset</span><span class="o">.</span><span class="n">cast</span><span class="p">(%</span><span class="no">DateRange</span><span class="p">{},</span>
<span class="p">%{</span><span class="ss">start:</span> <span class="sd">"</span><span class="s2">2/9/2016"</span><span class="p">,</span> <span class="k">end</span><span class="p">:</span> <span class="sd">"</span><span class="s2">5/9/2016"</span><span class="p">},</span>
<span class="err">~</span><span class="n">w</span><span class="p">(</span><span class="n">start</span> <span class="k">end</span><span class="p">),</span> <span class="err">~</span><span class="n">w</span><span class="p">())</span>
<span class="n">assert</span> <span class="n">changeset</span><span class="o">.</span><span class="n">errors</span> <span class="o">==</span> <span class="p">[]</span>
<span class="k">end</span></code></pre></figure>
<p>I’ve changed the <code class="highlighter-rouge">assert</code> from checking the <code class="highlighter-rouge">valid?</code> flag to asserting that there are no errors. This has the affect of printing the errors on failure:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test it can build a changeset (DateRangesTest)
test/date_ranges_test.exs:10
Assertion with == failed
code: changeset.errors() == []
lhs: [start: "is invalid", end: "is invalid"]
rhs: []
stacktrace:
test/date_ranges_test.exs:14
</code></pre>
</div>
<p>Now, this is a little more helpful.</p>
<p>So we are not casting <code class="highlighter-rouge">start</code> and <code class="highlighter-rouge">end</code> properly. The first problem is that we need to update our schema to use our new type. Let’s make this change:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gu">@@ -3,8 +3,8 @@ defmodule DateRanges.DateRange do
</span> import Ecto.Query, only: [from: 1, from: 2]
schema "date_ranges" do
<span class="gd">- field :start, Ecto.Date
- field :end, Ecto.Date
</span><span class="gi">+ field :start, DateRanges.CustomDate
+ field :end, DateRanges.CustomDate
</span>
timestamps
<span class="err"> end</span></code></pre></figure>
<p>And with this change the test passes!</p>
<h2 id="next-steps">Next Steps</h2>
<p>For me, the next step will be to integrate this custom type and its tests into my Phoenix based project. For you, I hope you’ll take what we learned here and look to see where you can apply it in your own Ecto and Phoenix based projects.</p>
<p><a href="http://learningelixir.joekain.com/custom-types-in-ecto/">Read more...</a></p>
http://learningelixir.joekain.com/fragments-in-ecto2016-02-02T00:00:00-08:002016-02-02T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>A few weeks ago I wrote about a <a href="/experiments-with-ecto-queries/">set of experiments I ran to try to diagnose a problem I was having with Ecto and timex</a>. I have still not figured out what’s going wrong but I’ve moved on a bit. In this post I want to write a little bit about Ecto and date ranges. I’ll use the date ranges as an example while describing how to use custom SQL in Ecto queries.</p>
<p>In this post I’ll be using Elixir 1.2.1 and the following dependencies (according to mix.lock):</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="p">%{</span><span class="sd">"</span><span class="s2">connection"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:connection</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.0.2"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">db_connection"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:db_connection</span><span class="p">,</span> <span class="sd">"</span><span class="s2">0.2.3"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">decimal"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:decimal</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.1.1"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">ecto"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:ecto</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.1.3"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">poolboy"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:poolboy</span><span class="p">,</span> <span class="sd">"</span><span class="s2">1.5.1"</span><span class="p">},</span>
<span class="sd">"</span><span class="s2">postgrex"</span><span class="p">:</span> <span class="p">{</span><span class="ss">:hex</span><span class="p">,</span> <span class="ss">:postgrex</span><span class="p">,</span> <span class="sd">"</span><span class="s2">0.11.0"</span><span class="p">}}</span></code></pre></figure>
<h2 id="date-ranges">Date Ranges</h2>
<p>Let’s start with a new mix project:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix new date_ranges
* creating README.md
* creating .gitignore
* creating mix.exs
* creating config
* creating config/config.exs
* creating lib
* creating lib/date_ranges.ex
* creating test
* creating test/test_helper.exs
* creating test/date_ranges_test.exs
Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:
cd date_ranges
mix test
Run "mix help" for more commands.
</code></pre>
</div>
<h2 id="setup-ecto">Setup Ecto</h2>
<p>I’ve covered this in <a href="/setting-up-ecto-in-elixir/">Setting up Ecto in Elixir</a> and repeated a lot of it in <a href="/experiments-with-ecto-queries/">Experiments with Ecto Queries</a> so I won’t repeat all the steps here. But here’s a summary of what I’ve done:</p>
<ul>
<li>Install and Start Ecto</li>
<li>Configure the Database</li>
<li>Write the Repo Module</li>
<li>Create the database</li>
</ul>
<p>These steps come directly from <a href="/setting-up-ecto-in-elixir/">Setting up Ecto in Elixir</a>. For more detail see that post or for to see the exact changes I made you can check out the <a href="https://github.com/joekain/date_ranges">github repo</a>.</p>
<h2 id="create-a-model">Create a Model</h2>
<p>Next we need a model to store some date ranges. We can write up something simple like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">DateRanges</span><span class="o">.</span><span class="no">DateRange</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Schema</span>
<span class="n">schema</span> <span class="sd">"</span><span class="s2">date_ranges"</span> <span class="k">do</span>
<span class="n">field</span> <span class="ss">:start</span><span class="p">,</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span>
<span class="n">field</span> <span class="ss">:end</span><span class="p">,</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>We just describe a date range with a <code class="highlighter-rouge">start</code> and <code class="highlighter-rouge">end</code> date. We also need a migration to add a corresponding table to our database. This will do:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">DateRanges</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">AddDateRanges</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="k">def</span> <span class="n">change</span> <span class="k">do</span>
<span class="n">create</span> <span class="n">table</span><span class="p">(</span><span class="ss">:date_ranges</span><span class="p">)</span> <span class="k">do</span>
<span class="n">add</span> <span class="ss">:start</span><span class="p">,</span> <span class="ss">:date</span>
<span class="n">add</span> <span class="ss">:end</span><span class="p">,</span> <span class="ss">:date</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>And then we can migrate our database:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix ecto.migrate
08:46:48.270 [info] == Running DateRanges.Repo.Migrations.AddDateRanges.change/0 forward
08:46:48.270 [info] create table date_ranges
08:46:48.277 [info] == Migrated in 0.0s
</code></pre>
</div>
<h2 id="querying-date-ranges">Querying Date Ranges</h2>
<p>My goal with this project is to show how to write Ecto queries against date ranges. So, let’s get started with a test to demonstrate:</p>
<h3 id="setup-tests">Setup tests</h3>
<p>We want to run our tests within a database transaction. In order to do this we need to configure Ecto to use the Sandbox adapter. Usually we would do this only for the test env. But, in this example we really only care about the test env. There is no dev or prod. So I’ll modify config.exs and add this line:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"> config :date_ranges, DateRanges.Repo,
adapter: Ecto.Adapters.Postgres,
database: "date_ranges",
username: "postgres",
password: "postgres",
<span class="gi">+ pool: Ecto.Adapters.SQL.Sandbox
</span><span class="err">+</span></code></pre></figure>
<p>Next, I added this line to begin transactions when starting up the tests:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff">ExUnit.start()
<span class="gi">+
+ Ecto.Adapters.SQL.begin_test_transaction(DateRanges.Repo)
</span><span class="err">+</span></code></pre></figure>
<p>As the final setup step I wrote up a skeleton for my test file with a <code class="highlighter-rouge">setup/1</code> function:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">DateRangeTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="n">alias</span> <span class="no">DateRanges</span><span class="o">.</span><span class="no">Repo</span>
<span class="n">alias</span> <span class="no">DateRanges</span><span class="o">.</span><span class="no">DateRange</span>
<span class="n">setup</span> <span class="n">tags</span> <span class="k">do</span>
<span class="k">unless</span> <span class="n">tags</span><span class="p">[</span><span class="ss">:async</span><span class="p">]</span> <span class="k">do</span>
<span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">SQL</span><span class="o">.</span><span class="n">restart_test_transaction</span><span class="p">(</span><span class="no">DateRanges</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span> <span class="p">[])</span>
<span class="k">end</span>
<span class="n">seed</span>
<span class="ss">:ok</span>
<span class="k">end</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">x"</span> <span class="k">do</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">seed</span> <span class="k">do</span>
<span class="n">ranges</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">1</span><span class="p">,</span> <span class="m">31</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">12</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">1</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">4</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">3</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">4</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">12</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">16</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">28</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">3</span><span class="p">,</span> <span class="m">10</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">3</span><span class="p">,</span> <span class="m">3</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">3</span><span class="p">,</span> <span class="m">7</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">1</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">9</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">6</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">12</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">21</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">25</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">5</span><span class="p">,</span> <span class="m">1</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">5</span><span class="p">,</span> <span class="m">5</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">5</span><span class="p">,</span> <span class="m">7</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">5</span><span class="p">,</span> <span class="m">8</span><span class="p">}</span> <span class="p">},</span>
<span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">5</span><span class="p">,</span> <span class="m">8</span><span class="p">},</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">5</span><span class="p">,</span> <span class="m">8</span><span class="p">}</span> <span class="p">},</span>
<span class="p">]</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">each</span> <span class="n">ranges</span><span class="p">,</span> <span class="k">fn</span> <span class="p">{</span><span class="n">s</span><span class="p">,</span> <span class="n">e</span><span class="p">}</span> <span class="o">-></span>
<span class="no">Repo</span><span class="o">.</span><span class="n">insert!</span> <span class="p">%</span><span class="no">DateRange</span><span class="p">{</span>
<span class="ss">start:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">from_erl</span><span class="p">(</span><span class="n">s</span><span class="p">),</span>
<span class="k">end</span><span class="p">:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">from_erl</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>The <code class="highlighter-rouge">setup</code> function restarts the database transaction at the beginning of each test. This maintains a clean environment for each tests. Then <code class="highlighter-rouge">setup</code> calls <code class="highlighter-rouge">seed</code> to fill in some default data that I reference in the tests.</p>
<p>The <code class="highlighter-rouge">seed</code> function uses a list, <code class="highlighter-rouge">ranges</code>, which is just a bunch of dates I made up (in Erlang date format). Then I insert them all into the database using <code class="highlighter-rouge">Enum.each</code> over <code class="highlighter-rouge">Repo.insert!</code>.</p>
<p>Finally, there is one test in the middle but it is currently a placeholder. The next step is to…</p>
<h3 id="write-a-test">Write a Test</h3>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">It can query overlapping ranges"</span> <span class="k">do</span>
<span class="n">target</span> <span class="o">=</span> <span class="p">%</span><span class="no">DateRange</span><span class="p">{</span>
<span class="ss">start:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">from_erl</span><span class="p">({</span><span class="m">2016</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">18</span><span class="p">}),</span>
<span class="k">end</span><span class="p">:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">from_erl</span><span class="p">({</span><span class="m">2016</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">22</span><span class="p">})</span>
<span class="p">}</span>
<span class="n">expected</span> <span class="o">=</span> <span class="p">%</span><span class="no">DateRange</span> <span class="p">{</span>
<span class="ss">start:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">from_erl</span><span class="p">({</span><span class="m">2016</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">21</span><span class="p">}),</span>
<span class="k">end</span><span class="p">:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="o">.</span><span class="n">from_erl</span><span class="p">({</span><span class="m">2016</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">25</span><span class="p">})</span>
<span class="p">}</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">target</span>
<span class="o">|></span> <span class="no">DateRange</span><span class="o">.</span><span class="n">overlapping</span>
<span class="o">|></span> <span class="no">Repo</span><span class="o">.</span><span class="n">one</span>
<span class="n">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">start</span> <span class="o">==</span> <span class="n">expected</span><span class="o">.</span><span class="n">start</span>
<span class="n">assert</span> <span class="n">result</span><span class="o">.</span><span class="k">end</span> <span class="o">==</span> <span class="n">expected</span><span class="o">.</span><span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Of course, this test fails:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1) test It can query overlapping ranges (DateRangeTest)
test/date_range_test.exs:17
** (UndefinedFunctionError) undefined function DateRanges.DateRange.overlapping/1
stacktrace:
(date_ranges) DateRanges.DateRange.overlapping(%DateRanges.DateRange{__meta__: #Ecto.Schema.Metadata<:built>, end: #Ecto.Date<2016-04-21>, id: nil, inserted_at: nil, start: #Ecto.Date<2016-04-18>, updated_at: nil})
test/date_range_test.exs:29
.
Finished in 0.1 seconds (0.1s on load, 0.03s on tests)
2 tests, 1 failure
</code></pre>
</div>
<h3 id="make-the-test-pass">Make the test pass</h3>
<p>To pass this test we need to implement an <code class="highlighter-rouge">overlapping/1</code> function. This function
is passed a target date and returns something we can pass to <code class="highlighter-rouge">Repo.all/2</code>. This means <code class="highlighter-rouge">DateRange.overlapping/1</code> must return an <code class="highlighter-rouge">Ecto.Query.t</code> value, that is it should be a query. We want something like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">overlapping</span><span class="p">(</span><span class="n">target</span><span class="p">)</span> <span class="k">do</span>
<span class="n">from</span> <span class="n">range</span> <span class="ow">in</span> <span class="no">DateRange</span><span class="p">,</span> <span class="ss">where:</span> <span class="n">overlaps</span><span class="p">(</span><span class="n">range</span><span class="p">,</span> <span class="o">^</span><span class="n">target</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>But what is <code class="highlighter-rouge">overlaps</code>? The Ecto query language doesn’t expose this.</p>
<p>Fortunately, <a href="http://www.postgresql.org/docs/9.0/static/functions-datetime.html">Postgres has an overlaps function</a>. We can write a custom SQL fragment to use this overlaps function like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="kn">import</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Query</span><span class="p">,</span> <span class="ss">only:</span> <span class="p">[</span><span class="ss">from:</span> <span class="m">1</span><span class="p">,</span> <span class="ss">from:</span> <span class="m">2</span><span class="p">]</span>
<span class="k">def</span> <span class="n">overlapping</span><span class="p">(</span><span class="n">target</span><span class="p">)</span> <span class="k">do</span>
<span class="n">from</span> <span class="n">range</span> <span class="ow">in</span> <span class="no">DateRanges</span><span class="o">.</span><span class="no">DateRange</span><span class="p">,</span> <span class="ss">where:</span> <span class="n">fragment</span><span class="p">(</span><span class="sd">"</span><span class="s2">(?, ?) OVERLAPS (?, ?)"</span><span class="p">,</span>
<span class="n">range</span><span class="o">.</span><span class="n">start</span><span class="p">,</span> <span class="n">range</span><span class="o">.</span><span class="k">end</span><span class="p">,</span> <span class="n">type</span><span class="p">(</span><span class="o">^</span><span class="n">target</span><span class="o">.</span><span class="n">start</span><span class="p">,</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="p">),</span> <span class="n">type</span><span class="p">(</span><span class="o">^</span><span class="n">target</span><span class="o">.</span><span class="k">end</span><span class="p">,</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Date</span><span class="p">))</span>
<span class="k">end</span></code></pre></figure>
<p>With this implementation of <code class="highlighter-rouge">overlapping/1</code> the test passes:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Finished in 0.1 seconds (0.1s on load, 0.03s on tests)
2 tests, 0 failures
</code></pre>
</div>
<p>How does this work?</p>
<p>The beginning of our query looks quite normal: <code class="highlighter-rouge">from range in DateRanges.DateRange, where:</code>. Next, we have a call to the <code class="highlighter-rouge">fragment/1</code> macro. It starts with a string:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="sd">"</span><span class="s2">(?, ?) OVERLAPS (?, ?)"</span></code></pre></figure>
<p>This is a fragment of SQL which uses the <code class="highlighter-rouge">OVERLAPS</code> function to compare two date ranges. The <code class="highlighter-rouge">?</code> characters are placeholders that need to be filled in with real values. Those values are described as further arguments to <code class="highlighter-rouge">fragment/1</code>. We are effectively generating SQL that would look something like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="sd">"</span><span class="s2">(range.start, range.end) OVERLAPS (^target.start, target.end)"</span></code></pre></figure>
<p>The last piece to explain is the the <code class="highlighter-rouge">type/2</code> calls. With fragments Ecto doesn’t know the type of the data elements and we can use the <code class="highlighter-rouge">type/2</code> function to describe the type. In <code class="highlighter-rouge">overlapping/1</code> we describe <code class="highlighter-rouge">target.start</code> and <code class="highlighter-rouge">target.end</code> as <code class="highlighter-rouge">Ecto.Date</code>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In this post we looked at how to use Ecto fragments to inject custom SQL into a composable query. I learned a lot about how to do this while working on an app of mine. And I hope I was able to explain it and help others to do the same thing.</p>
<p><a href="http://learningelixir.joekain.com/fragments-in-ecto/">Read more...</a></p>
http://learningelixir.joekain.com/use-import-require-in-elixir2016-01-20T00:00:00-08:002016-01-20T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>There a several special forms for referring to other modules in Elixir. These include:</p>
<ul>
<li><code class="highlighter-rouge">use</code></li>
<li><code class="highlighter-rouge">import</code></li>
<li><code class="highlighter-rouge">require</code></li>
<li><code class="highlighter-rouge">alias</code></li>
</ul>
<p>These each have their own meaning but until you get used to them it can be hard to know which one to use.</p>
<p>To demonstrate each of these special forms let’s create a new test project.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix new use_import_require
* creating README.md
* creating .gitignore
* creating mix.exs
* creating config
* creating config/config.exs
* creating lib
* creating lib/use_import_require.ex
* creating test
* creating test/test_helper.exs
* creating test/use_import_require_test.exs
Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:
cd use_import_require
mix test
Run "mix help" for more commands.
</code></pre>
</div>
<p>This comes with a brand new empty module:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UseImportRequire</span> <span class="k">do</span>
<span class="k">end</span></code></pre></figure>
<p>Though the course of this post we’ll fill in and make use of <code class="highlighter-rouge">use</code>, <code class="highlighter-rouge">import</code>, <code class="highlighter-rouge">require</code>, and <code class="highlighter-rouge">alias</code>.</p>
<h2 id="elixir-alias">Elixir alias</h2>
<p>I’ll start with <code class="highlighter-rouge">alias</code> whose only purpose is to make it easier to type (and maybe read) code. To make an alias we’ll need another module. Let’s add one:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="c1"># lib/use_import_require/alias_me.ex</span>
<span class="k">defmodule</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">AliasMe</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">function</span> <span class="k">do</span>
<span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="sd">"</span><span class="si">#{</span><span class="bp">__MODULE__</span><span class="si">}</span><span class="s2">.function"</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Now, if we want to call this <code class="highlighter-rouge">function</code> from <code class="highlighter-rouge">UseImportRequire</code> we would have to reference with its fully qualified name: <code class="highlighter-rouge">UseImportRequire.AliasMe.function</code>. But that’s a lot to write so we can use <code class="highlighter-rouge">alias</code> like this and use a shorter form of the name:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UseImportRequire</span> <span class="k">do</span>
<span class="n">alias</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">AliasMe</span>
<span class="k">def</span> <span class="n">alias_test</span> <span class="k">do</span>
<span class="no">AliasMe</span><span class="o">.</span><span class="n">function</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>And that’s it. That’s what <code class="highlighter-rouge">alias</code> does, it let’s you use shorter names. Of course, there is some more to the syntax like aliasing more than one module at a time. And you have some flexibility about how many nested modules you alias away. But in term of purpose, this is it: <code class="highlighter-rouge">alias</code> shortens the name.</p>
<p>There are of course some tradeoffs to consider with <code class="highlighter-rouge">alias</code>. In the simple example above there’s not much to consider. Most likely everything in this app will start with the <code class="highlighter-rouge">UseImportRequire</code> so omitting it has little cost and makes code more readable by eliminating noise. However, as the number of modules in your project grows you may end up with multiple modules with the same leaf module name. Using alias you might end up with code that’s confusing to the reader. You should ask yourself, if I alias am I going to create any ambiguity?</p>
<p><em>Edit Jan 23, 2016:</em> As Philip Claren pointed out in the comments, it is also possible to pick the name for your alias using the alias-as form. Adding on to our example:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UseImportRequire</span> <span class="k">do</span>
<span class="n">alias</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">AliasMe</span>
<span class="n">alias</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">AliasMe</span><span class="p">,</span> <span class="ss">as:</span> <span class="no">AnotherName</span>
<span class="k">def</span> <span class="n">alias_test</span> <span class="k">do</span>
<span class="no">AliasMe</span><span class="o">.</span><span class="n">function</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">alias_as_test</span> <span class="k">do</span>
<span class="no">AnotherName</span><span class="o">.</span><span class="n">function</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>We see we can add an <code class="highlighter-rouge">as: name</code> parameter to <code class="highlighter-rouge">alias</code> to set any aliased name we desire. I’ve picked <code class="highlighter-rouge">AnotherName</code> in the example.</p>
<p>Alias-as can be really helpful if you have two modules with the same leaf-name. For example, if you had <code class="highlighter-rouge">UseImportRequire.FirstKind.AliasMe</code> and <code class="highlighter-rouge">UseImportRequire.SecondKind.AliasMe</code> then simply aliasing both modules wouldn’t work. There would be ambiguity. With alias-as you can choose a different name for one of the modules.</p>
<h2 id="elixir-import">Elixir import</h2>
<p>Now, suppose you still think that <code class="highlighter-rouge">AliasMe.function</code> is too much to write. You could use <code class="highlighter-rouge">import</code>. For an import example I’ll create a module to import:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="c1"># lib/use_import_require/import_me.ex</span>
<span class="k">defmodule</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">ImportMe</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">function</span> <span class="k">do</span>
<span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="sd">"</span><span class="si">#{</span><span class="bp">__MODULE__</span><span class="si">}</span><span class="s2">.function"</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>and, I’ll reference this from <code class="highlighter-rouge">UseImportRequire</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UseImportRequire</span> <span class="k">do</span>
<span class="n">alias</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">AliasMe</span>
<span class="kn">import</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">ImportMe</span>
<span class="k">def</span> <span class="n">alias_test</span> <span class="k">do</span>
<span class="no">AliasMe</span><span class="o">.</span><span class="n">function</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">import_test</span> <span class="k">do</span>
<span class="n">function</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>So, now we can reference the function as just <code class="highlighter-rouge">function</code>. Nice an short. But at what cost does this come?</p>
<p>It is much easier to create ambiguity in this case. For example, if we imported both the <code class="highlighter-rouge">AliasMe</code> and <code class="highlighter-rouge">ImportMe</code> modules we’d end up with two functions named <code class="highlighter-rouge">function</code> and in fact would not compile. Note that the compilation error is lazy in that it would fail unless an ambiguously named function is actually called.</p>
<p>I would recommend using <code class="highlighter-rouge">import</code> sparingly. It removes a lot of information which can be a burden for any reader of your code. However, there are a few cases where import is helpful. If you are writing a module that is very focused in that it makes heavy use of a specific module then <code class="highlighter-rouge">import</code> may make sense. One common example is that in a module that makes extensive use of Ecto queries it is common to import <code class="highlighter-rouge">Ecto.Query</code>.</p>
<p>The <code class="highlighter-rouge">import</code> macro also allows importing of specific functions or macros. This limits “namespace pollution” and can reduce the chance of ambiguity or confusion. Again, this is common with <code class="highlighter-rouge">Ecto.Query</code> - the documentation recommends:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="kn">import</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Query</span><span class="p">,</span> <span class="ss">only:</span> <span class="p">[</span><span class="ss">from:</span> <span class="m">2</span><span class="p">]</span></code></pre></figure>
<p>in order to import only the <code class="highlighter-rouge">Ecto.Query.from/2</code> macro.</p>
<h2 id="elixir-require">Elixir require</h2>
<p>The <code class="highlighter-rouge">require</code> macro instructs the compiler to load the specified module before compiling the containing module. This is only necessary if you want to reference macros from the specified module. For example, we would need:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UseImportRequire</span> <span class="k">do</span>
<span class="kn">require</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">RequireMe</span>
<span class="k">end</span></code></pre></figure>
<p>if the <code class="highlighter-rouge">RequireMe</code> module contained a macro we wanted to use. Nothing special is done to the macro name. We would still need to reference it with its fully qualified name.</p>
<p>In this way <code class="highlighter-rouge">require</code> and <code class="highlighter-rouge">import</code> have some overlapping purpose. Either can be used to access macros in other modules. Though their effects on the namespace differ.</p>
<h2 id="elixir-use">Elixir use</h2>
<p>The <code class="highlighter-rouge">use</code> macro invokes a special macro, called <code class="highlighter-rouge">__using__/1</code>, from the specified module. Here’s an example:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="c1"># lib/use_import_require/use_me.ex</span>
<span class="k">defmodule</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">UseMe</span> <span class="k">do</span>
<span class="k">defmacro</span> <span class="n">__using__</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="k">do</span>
<span class="kn">quote</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">use_test</span> <span class="k">do</span>
<span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="sd">"</span><span class="s2">use_test"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>and we add this line to <code class="highlighter-rouge">UseImportRequire</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="kn">use</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">UseMe</span></code></pre></figure>
<p>Using <code class="highlighter-rouge">UseImportRequire.UseMe</code> defines a <code class="highlighter-rouge">use_test/0</code> function through invocation of the <code class="highlighter-rouge">__using__/1</code> macro.</p>
<p>This is all that <code class="highlighter-rouge">use</code> does. However, it is common for the <code class="highlighter-rouge">__using__</code> macro to in turn call <code class="highlighter-rouge">alias</code>, <code class="highlighter-rouge">require</code>, or <code class="highlighter-rouge">import</code>. This in turn will create aliases or imports in the using module. This allows the module being used to define a policy for how its functions and macros should be referenced. This can be quite flexible in that <code class="highlighter-rouge">__using__/1</code> may set up references to other modules, especially submodules.</p>
<p>The Phoenix framework makes use of <code class="highlighter-rouge">use</code> and <code class="highlighter-rouge">__using__/1</code> to cut down on the need for repetitive <code class="highlighter-rouge">alias</code> and <code class="highlighter-rouge">import</code> calls in user defined modules.</p>
<p>Here’s an nice and short example from the <code class="highlighter-rouge">Ecto.Migration</code> module:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmacro</span> <span class="n">__using__</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="k">do</span>
<span class="kn">quote</span> <span class="ss">location:</span> <span class="ss">:keep</span> <span class="k">do</span>
<span class="kn">import</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="nv">@disable_ddl_transaction</span> <span class="no">false</span>
<span class="nv">@before_compile</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>The <code class="highlighter-rouge">Ecto.Migration.__using__/1</code> macro includes an <code class="highlighter-rouge">import</code> call so that if <code class="highlighter-rouge">use Ecto.Migration</code> you also <code class="highlighter-rouge">import Ecto.migration</code>. It also sets up a module property which I assume control Ecto’s behavior.</p>
<p>To recap: the <code class="highlighter-rouge">use</code> macro just invokes the <code class="highlighter-rouge">__using__/1</code> macro of the specified module. To really understand what that does you need to read the <code class="highlighter-rouge">__using__/1</code> macro.</p>
<h2 id="referencing-modules">Referencing Modules</h2>
<p>Now, which of the above macros should you use if you just want to call functions from another module? The answer is: none. Instead, you can just reference the functions directly. Here’s an example module we can reference:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="c1"># lib/use_import_require/reference_me.ex</span>
<span class="k">defmodule</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">ReferenceMe</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">function</span> <span class="k">do</span>
<span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="sd">"</span><span class="si">#{</span><span class="bp">__MODULE__</span><span class="si">}</span><span class="s2">.function"</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>and here is how we can access the function:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">reference_test</span> <span class="k">do</span>
<span class="no">UseImportRequire</span><span class="o">.</span><span class="no">ReferenceMe</span><span class="o">.</span><span class="n">function</span>
<span class="k">end</span></code></pre></figure>
<p>That’s it. We don’t need to <code class="highlighter-rouge">use</code>, <code class="highlighter-rouge">import</code>, or <code class="highlighter-rouge">require</code> the module. We just use the fully qualified name.</p>
<h2 id="scope">Scope</h2>
<p>As I’ve mentioned there are tradeoffs for using <code class="highlighter-rouge">alias</code> and <code class="highlighter-rouge">import</code> between convenience and clarity. There is another way to help mitigate this tradeoff. The <code class="highlighter-rouge">alias</code> and <code class="highlighter-rouge">import</code> macros don’t need to be called at the outer module scope as we have been using them. They can, for example, be called from within another function. Here’s an example using <code class="highlighter-rouge">import</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">WithScope</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">scope_test</span> <span class="k">do</span>
<span class="kn">import</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">ReferenceMe</span>
<span class="n">function</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>I’ve had to add this to a new module rather than <code class="highlighter-rouge">UseImportRequire</code> due to a problem with ambiguity.</p>
<p>Using this scoped version we can’t access <code class="highlighter-rouge">function</code> without a qualified name in another function. for example, this won’t compile:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">WithScope</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">scope_test</span> <span class="k">do</span>
<span class="kn">import</span> <span class="no">UseImportRequire</span><span class="o">.</span><span class="no">ReferenceMe</span>
<span class="n">function</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">failing_scope_test</span> <span class="k">do</span>
<span class="n">function</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>The <code class="highlighter-rouge">alias</code> macro can be used within a function in the same way.</p>
<p>Calling <code class="highlighter-rouge">import</code> and <code class="highlighter-rouge">alias</code> within a narrow scope this way helps limit the amount of “namespace pollution” that happens in your code.</p>
<p>Also, using a narrow scope this way makes the code a little clearer by keeping the <code class="highlighter-rouge">alias</code> or <code class="highlighter-rouge">import</code> closer to the reference thereby making it easier for the reader to keep track of what’s going on. This can be especially useful as your modules start to get longer.</p>
<h2 id="an-example">An Example</h2>
<p>Here’s a really nice example of using <code class="highlighter-rouge">import</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Orthrus</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">CreateUser</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="k">def</span> <span class="n">change</span> <span class="k">do</span>
<span class="n">create</span> <span class="n">table</span><span class="p">(</span><span class="ss">:users</span><span class="p">)</span> <span class="k">do</span>
<span class="n">add</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">add</span> <span class="ss">:username</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">add</span> <span class="ss">:password_hash</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">add</span> <span class="ss">:email</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>The <code class="highlighter-rouge">use Ecto.Migration</code> call invokes <code class="highlighter-rouge">Ecto.Migration.__using__/1</code>. And we saw above that this macro in turn calls <code class="highlighter-rouge">import Ecto.Migration</code>. The import allows us to write very clean code in the migration. We can call <code class="highlighter-rouge">create</code>, <code class="highlighter-rouge">add</code>, <code class="highlighter-rouge">timestamps</code> without needing to clutter up the code with an <code class="highlighter-rouge">Ecto.Migration</code> prefix.</p>
<p>For migrations, this is a good tradeoff a migration is narrowly focused task. When you read these references to <code class="highlighter-rouge">create table</code>, and <code class="highlighter-rouge">add</code> you are in the mindset of thinking about database migrations so this code makes sense.</p>
<p>If you have other tasks that are not as focused you may want to ask yourself if <code class="highlighter-rouge">import</code> is the right choice.</p>
<p><a href="http://learningelixir.joekain.com/use-import-require-in-elixir/">Read more...</a></p>
http://learningelixir.joekain.com/installing-multiple-elixir-version-with-kiex2016-01-15T00:00:00-08:002016-01-15T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>I like to try out different versions of Elixir including release candidates . This can be especially useful when I want to try out new features and write about them before the features make it into an official Elixir release. An example of this would be last month’s post on the <a href="/learning-elixir-with/">Elixir’s new Special form “with”</a></p>
<p>The tool I use to manage multiple versions of Elixir is <a href="http://taylor.github.io/kiex/">Kiex</a>.</p>
<h2 id="using-kiex">Using Kiex</h2>
<p>Kiex’s documentation is quite good and you should refer to it for the basic installation.</p>
<p>The commands I use most frequently are:</p>
<ul>
<li><code class="highlighter-rouge">kiex list known</code> - to list all the Elixir versions Kiex knows about:</li>
<li><code class="highlighter-rouge">kiex install <version></code> to install Elixir version <code class="highlighter-rouge"><version></code> on my system.</li>
<li><code class="highlighter-rouge">kiex use <version></code> to setup my current shell to use Elixir version <code class="highlighter-rouge"><version></code>.</li>
</ul>
<p>So for example, right now the known versions are:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>kiex list known
Getting the available releases from https://github.com/elixir-lang/elixir/releases
Known Elixir releases:
1.2.1
1.2.0
1.2.0-rc.1
1.2.0-rc.0
1.1.1
1.1.0
1.1.0-rc.0
0.9.3
0.9.2
0.9.1
0.9.0
0.8.3
0.8.2
0.8.1
0.8.0
0.7.2
1.0.5
1.0.4
1.0.3
1.0.2
1.0.1
1.0.0
1.0.0-rc2
1.0.0-rc1
0.15.1
0.15.0
0.14.3
0.14.2
0.14.1
0.14.0
</code></pre>
</div>
<p>There are plenty of versions to choose from including release candidates. And, Elixir 1.2.1 was released yesterday! I’ll install it like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ kiex install 1.2.1
Switched to branch 'master'
Your branch is behind 'origin/master' by 289 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
remote: Counting objects: 394, done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 394 (delta 202), reused 183 (delta 183), pack-reused 162
Receiving objects: 100% (394/394), 123.77 KiB | 0 bytes/s, done.
Resolving deltas: 100% (245/245), completed with 76 local objects.
From https://github.com/elixir-lang/elixir
ad03436..a43e6fc master -> origin/master
bcc92cc..a3d88dc v1.2 -> origin/v1.2
* [new tag] v1.2.1 -> v1.2.1
Switched to a new branch 'v1.2.1'
From https://github.com/elixir-lang/elixir
* tag v1.2.1 -> FETCH_HEAD
Already up-to-date.
cd lib/elixir && "/Users/jkain/.kiex/builds/elixir-git/rebar" clean
==> elixir (clean)
...
Installed Elixir version 1.2.1
Load with:
kiex use 1.2.1
or load the elixir environment file with:
source $HOME/.kiex/elixirs/elixir-1.2.1.env
</code></pre>
</div>
<p>Now that it’s installed I can set up my shell to run it like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ kiex use 1.2.1
Using 1.2.1
$ iex
Erlang/OTP 18 [erts-7.2] [source-e6dd627] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]
Interactive Elixir (1.2.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
</code></pre>
</div>
<h2 id="bash-prompt">Bash Prompt</h2>
<p>I like to be able to quickly tell which version of Elixir I’m using so I include the version number in my Bash prompt. Kiex is kind enough to set the environment variable <code class="highlighter-rouge">ELIXIR_VERSION</code>. I can use that variable in my prompt like this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">RED</span><span class="o">=</span><span class="s2">"</span><span class="se">\[\0</span><span class="s2">33[0;31m</span><span class="se">\]</span><span class="s2">"</span>
<span class="nv">LIGHT_BLUE</span><span class="o">=</span><span class="s2">"</span><span class="se">\[\0</span><span class="s2">33[1;34m</span><span class="se">\]</span><span class="s2">"</span>
<span class="nv">DEFAULT</span><span class="o">=</span><span class="s2">"</span><span class="se">\[\0</span><span class="s2">33[0m</span><span class="se">\]</span><span class="s2">"</span>
<span class="k">function </span>ex_version <span class="o">{</span>
<span class="k">if</span> <span class="o">[</span> -n <span class="s2">"</span><span class="nv">$ELIXIR_VERSION</span><span class="s2">"</span> <span class="o">]</span>; <span class="k">then
</span><span class="nb">echo</span> <span class="s2">":ex-</span><span class="nv">$ELIXIR_VERSION</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="o">}</span>
<span class="nv">PS1</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">RED</span><span class="k">}</span><span class="s2">jkain-mbp</span><span class="k">${</span><span class="nv">LIGHT_BLUE</span><span class="k">}</span><span class="se">\$</span><span class="s2">(ex_version)</span><span class="k">${</span><span class="nv">DEFAULT</span><span class="k">}</span><span class="s2"> </span><span class="se">\w\n</span><span class="nv">$ </span><span class="s2">"</span></code></pre></figure>
<p>And this is what my prompt looks like:</p>
<p><img src="/images/my_prompt.png" alt="My bash prompt" /></p>
<h2 id="conclusion">Conclusion</h2>
<p>This was just a brief post to describe how I make use of Kiex and what you can do with it. I hope you’ve found this helpful.</p>
<p><a href="http://learningelixir.joekain.com/installing-multiple-elixir-version-with-kiex/">Read more...</a></p>
http://learningelixir.joekain.com/experiments-with-ecto-queries2016-01-08T00:00:00-08:002016-01-08T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>Readers, thank you all for your patience as I haven’t posted for several weeks.</p>
<p>Last time I wrote about <a href="/setting-up-ecto-in-elixir/">setting up Ecto from scratch</a> and I promised to continue with this topic. However, this post will be another topice though it is Ecto related. I’m going to go through a series of troubleshooting experiments in order to learn more about a problem I’ve run into in one of my private projects.</p>
<p>In this post I’m using</p>
<ul>
<li>Elixir 1.2.0</li>
<li>Ecto 1.1.1</li>
<li>timex 1.0.0</li>
<li>timex_ecto 0.7.0</li>
</ul>
<h2 id="the-models">The Models</h2>
<p>In my private project I have a Phoenix application with a model that represents an object that is available during specific date/time ranges. We’ll call this <code class="highlighter-rouge">Object</code> for now. There is a second model, <code class="highlighter-rouge">AvailableDateRange</code> which represents a single range of dates. An <code class="highlighter-rouge">Object</code> has many <code class="highlighter-rouge">AvailableDateRange</code>s.</p>
<p>I’m using Timex to describe the date/times.</p>
<p>Here’s a stripped down version of these structures:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Project</span><span class="o">.</span><span class="no">Object</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Project</span><span class="o">.</span><span class="no">Web</span><span class="p">,</span> <span class="ss">:model</span>
<span class="kn">require</span> <span class="no">Logger</span>
<span class="n">schema</span> <span class="sd">"</span><span class="s2">object"</span> <span class="k">do</span>
<span class="n">field</span> <span class="ss">:title</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">field</span> <span class="ss">:description</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">belongs_to</span> <span class="ss">:user</span><span class="p">,</span> <span class="no">User</span>
<span class="n">has_many</span> <span class="ss">:available_date_ranges</span><span class="p">,</span> <span class="no">Project</span><span class="o">.</span><span class="no">AvailableDateRange</span><span class="p">,</span> <span class="ss">on_replace:</span> <span class="ss">:delete</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">defmodule</span> <span class="no">Project</span><span class="o">.</span><span class="no">AvailableDateRange</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Project</span><span class="o">.</span><span class="no">Web</span><span class="p">,</span> <span class="ss">:model</span>
<span class="kn">use</span> <span class="no">Timex</span><span class="o">.</span><span class="no">Ecto</span><span class="o">.</span><span class="no">Timestamps</span>
<span class="n">schema</span> <span class="sd">"</span><span class="s2">available_date_ranges"</span> <span class="k">do</span>
<span class="n">field</span> <span class="ss">:start</span><span class="p">,</span> <span class="no">Timex</span><span class="o">.</span><span class="no">Ecto</span><span class="o">.</span><span class="no">DateTimeWithTimezone</span>
<span class="n">field</span> <span class="ss">:end</span><span class="p">,</span> <span class="no">Timex</span><span class="o">.</span><span class="no">Ecto</span><span class="o">.</span><span class="no">DateTimeWithTimezone</span>
<span class="n">belongs_to</span> <span class="ss">:object</span><span class="p">,</span> <span class="no">Project</span><span class="o">.</span><span class="no">Object</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<h2 id="the-query">The Query</h2>
<p>In my project I have a query that looks like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">available_for</span><span class="p">(</span><span class="n">query</span> <span class="p">\\</span> <span class="no">Project</span><span class="o">.</span><span class="no">Object</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span> <span class="k">do</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">parse_date</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
<span class="n">e</span> <span class="o">=</span> <span class="n">parse_date</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="n">from</span> <span class="n">object</span> <span class="ow">in</span> <span class="n">query</span><span class="p">,</span> <span class="ss">join:</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">assoc</span><span class="p">(</span><span class="n">object</span><span class="p">,</span> <span class="ss">:available_date_ranges</span><span class="p">),</span> <span class="ss">where:</span> <span class="n">a</span><span class="o">.</span><span class="n">start</span> <span class="o">></span> <span class="o">^</span><span class="n">s</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">parse_date</span><span class="p">(</span><span class="n">date</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">timex</span><span class="p">}</span> <span class="o">=</span> <span class="no">Timex</span><span class="o">.</span><span class="no">DateFormat</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">date</span><span class="p">,</span> <span class="sd">"</span><span class="s2">{M}/{D}/{YYYY}"</span><span class="p">)</span>
<span class="n">timex</span>
<span class="k">end</span></code></pre></figure>
<p>The function <code class="highlighter-rouge">available_for/3</code> is a <a href="http://blog.drewolson.org/composable-queries-ecto/">composable query</a>. I can pass any query that returns <code class="highlighter-rouge">Object</code> records int <code class="highlighter-rouge">available_for/3</code> and it builds a new query that limits the original query to those <code class="highlighter-rouge">Object</code> records that are available for a specific date. The function is unfinished. It checks only the starting date and not the ending date. I haven’t finished this because it isn’t working.</p>
<h2 id="the-problem">The Problem</h2>
<p>When I try to use this query I get this error:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>** (exit) an exception was raised:
** (FunctionClauseError) no function clause matching in :lists.zip/2
(stdlib) lists.erl:386: :lists.zip([{ {2016, 1, 5}, {0, 0, 0, 0}}, "UTC"], [])
(postgrex) lib/postgrex/extensions/binary.ex:297: Postgrex.Extensions.Binary.encode_record/3
(postgrex) lib/postgrex/protocol.ex:299: anonymous fn/2 in Postgrex.Protocol.encode_params/1
(elixir) lib/enum.ex:1043: anonymous fn/3 in Enum.map/2
(elixir) lib/enum.ex:1387: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:1043: Enum.map/2
(postgrex) lib/postgrex/protocol.ex:294: Postgrex.Protocol.encode_params/1
(postgrex) lib/postgrex/protocol.ex:264: Postgrex.Protocol.send_params/2
(postgrex) lib/postgrex/protocol.ex:136: Postgrex.Protocol.message/3
(postgrex) lib/postgrex/connection.ex:417: Postgrex.Connection.new_data/2
(postgrex) lib/postgrex/connection.ex:292: Postgrex.Connection.handle_info/2
(stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
(stdlib) gen_server.erl:681: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
</code></pre>
</div>
<p>This error happens deep inside Postgrex, the Postgres adapter for Ecto. I can understand that the <code class="highlighter-rouge">zip</code> function doesn’t match on lists of different sizes but I don’t know why this happening. In the rest of the post we will investigate what is going wrong and how we can make this work.</p>
<h2 id="experiments">Experiments</h2>
<p>I will approach this investigation as a series of experiments, each experiment building on the previous. I’ll start with a very simple Ecto program and will build up toward something like the example from my private project until I can reproduce this same error.</p>
<p>The goals of these are experiments are to</p>
<ul>
<li>Build up an understanding of what’s going on with my query.</li>
<li>Build a minimal reproduction case that I can debug or send off to the Ecto developers for further investigation.</li>
</ul>
<h3 id="experiment-1---setup-a-basic-ecto-application">Experiment 1 - Setup a basic Ecto application</h3>
<p>First I need a basic Elixir application with Ecto. I covered exactly this in the last post so I won’t go over it in detail here. But here are the commands I’m running so you can follow along.</p>
<p>First, create a new supervised mix project. EE is “Ecto Experiments”</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="gp">$ </span>mix new --sup ee</code></pre></figure>
<p>Next, I put everything under source control.</p>
<p>Install ecto with postgrex and start the Ecto application. I can do this in mix.exs.</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gu">@@ -14,7 +14,7 @@ defmodule EE.Mixfile do
</span> #
# Type "mix help compile.app" for more information
def application do
<span class="gd">- [applications: [:logger]]
</span><span class="gi">+ [applications: [:logger, :postgrex, :ecto]]
</span> end
# Dependencies can be Hex packages:
<span class="gu">@@ -27,6 +27,9 @@ defmodule EE.Mixfile do
</span> #
# Type "mix help deps" for more examples and options
defp deps do
<span class="gd">- []
</span><span class="gi">+ [
+ {:postgrex, ">= 0.0.0"},
+ {:ecto, "~> 1.0"},
+ ]
</span> end
<span class="err"> end</span></code></pre></figure>
<p>I then configured the database by adding the following to config/config.exs:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">config</span> <span class="ss">:ee</span><span class="p">,</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">adapter:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">Postgres</span><span class="p">,</span>
<span class="ss">database:</span> <span class="sd">"</span><span class="s2">ecto_experiments"</span><span class="p">,</span>
<span class="ss">username:</span> <span class="sd">"</span><span class="s2">postgres"</span><span class="p">,</span>
<span class="ss">password:</span> <span class="sd">"</span><span class="s2">postgres"</span></code></pre></figure>
<p>Then, I created a Repo in lib/ee/repo.ex:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">otp_app:</span> <span class="ss">:ee</span>
<span class="k">end</span></code></pre></figure>
<p>Then I added the Repo as a worker in my child specification:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gh">index ae71ede..7524c78 100644
</span><span class="gd">--- a/lib/ee.ex
</span><span class="gi">+++ b/lib/ee.ex
</span><span class="gu">@@ -8,7 +8,7 @@ defmodule Ee do
</span>
children = [
# Define workers and child supervisors to be supervised
<span class="gd">- # worker(Ee.Worker, [arg1, arg2, arg3]),
</span><span class="gi">+ worker(Ee.Repo, [])
</span> ]
<span class="err"> # See http://elixir-lang.org/docs/stable/elixir/Supervisor.html</span></code></pre></figure>
<p>and tested the whole thing with</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="gp">$ </span>mix ecto.create
The database <span class="k">for </span>Ee.Repo has been created.</code></pre></figure>
<p>I committed all the changes. The next step is to build a model. We’ll start with this as lib/ee/object.ex:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">EE</span><span class="o">.</span><span class="no">Object</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Schema</span>
<span class="n">schema</span> <span class="sd">"</span><span class="s2">object"</span> <span class="k">do</span>
<span class="n">field</span> <span class="ss">:title</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">field</span> <span class="ss">:description</span><span class="p">,</span> <span class="ss">:string</span>
<span class="c1"># has_many :available_date_ranges, Project.AvailableDateRange, on_replace: :delete</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>I’ve stripped this down slightly from the example I showed from my private project.</p>
<ul>
<li>I’ve removed <code class="highlighter-rouge">Project.Web</code> since we are not using Phoenix. Instead we <code class="highlighter-rouge">use Ecto.Schema</code>.</li>
<li>I’ve removed logger.</li>
<li>I’ve removed <code class="highlighter-rouge">belongs_to :user, User</code> since we haven’t built a <code class="highlighter-rouge">User</code> model.</li>
<li>I’ve commented out the <code class="highlighter-rouge">has_many</code> relationship. We’ll add it back when we add <code class="highlighter-rouge">AvailableDateRange</code>.</li>
</ul>
<p>Here’s the associated migration:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">EE</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">AddObject</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="k">def</span> <span class="n">change</span> <span class="k">do</span>
<span class="n">create</span> <span class="n">table</span><span class="p">(</span><span class="ss">:object</span><span class="p">)</span> <span class="k">do</span>
<span class="n">add</span> <span class="ss">:title</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">add</span> <span class="ss">:description</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>and I can run it like this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="gp">$ </span>mix ecto.migrate
14:53:37.623 <span class="o">[</span>info] <span class="o">==</span> Running Ee.Repo.Migrations.AddObject.change/0 forward
14:53:37.623 <span class="o">[</span>info] create table object
14:53:37.628 <span class="o">[</span>info] <span class="o">==</span> Migrated <span class="k">in </span>0.0s</code></pre></figure>
<p>We now have a basic setup so I committed everything. And I added a git tag called experiment/setup to mark this point for later reference.</p>
<h3 id="experiment-2---tests">Experiment 2 - Tests</h3>
<p>Next, let’s write a test and any additional required code to create an object and insert it into the database.</p>
<p>In the last experiment we configured the database for the dev environment. We’ll need to add a configuration for the test environment. We’ll also have to set things up so that we can maintain separate dev and test configs.</p>
<p>Mix generated support for this but commented out. In config/config.exs we have:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="c1"># It is also possible to import configuration files, relative to this</span>
<span class="c1"># directory. For example, you can emulate configuration per environment</span>
<span class="c1"># by uncommenting the line below and defining dev.exs, test.exs and such.</span>
<span class="c1"># Configuration from the imported file will override the ones defined</span>
<span class="c1"># here (which is why it is important to import them last).</span>
<span class="c1">#</span>
<span class="c1"># import_config "#{Mix.env}.exs"</span></code></pre></figure>
<p>We’ll uncomment this import and move our existing config to config/dev.exs:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="c1"># config/dev.exs</span>
<span class="kn">use</span> <span class="no">Mix</span><span class="o">.</span><span class="no">Config</span>
<span class="n">config</span> <span class="ss">:ee</span><span class="p">,</span> <span class="no">EE</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">adapter:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">Postgres</span><span class="p">,</span>
<span class="ss">database:</span> <span class="sd">"</span><span class="s2">ecto_experiments"</span><span class="p">,</span>
<span class="ss">username:</span> <span class="sd">"</span><span class="s2">postgres"</span><span class="p">,</span>
<span class="ss">password:</span> <span class="sd">"</span><span class="s2">postgres"</span></code></pre></figure>
<p>Then we create a new file config/test.exs:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="c1"># config/test.exs</span>
<span class="kn">use</span> <span class="no">Mix</span><span class="o">.</span><span class="no">Config</span>
<span class="n">config</span> <span class="ss">:ee</span><span class="p">,</span> <span class="no">EE</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">adapter:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">Postgres</span><span class="p">,</span>
<span class="ss">database:</span> <span class="sd">"</span><span class="s2">ecto_experiments_test"</span><span class="p">,</span>
<span class="ss">username:</span> <span class="sd">"</span><span class="s2">postgres"</span><span class="p">,</span>
<span class="ss">password:</span> <span class="sd">"</span><span class="s2">postgres"</span><span class="p">,</span>
<span class="ss">pool:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">SQL</span><span class="o">.</span><span class="no">Sandbox</span></code></pre></figure>
<p>The difference between the dev and test configs are:</p>
<ul>
<li><code class="highlighter-rouge">database:</code> We have separate databases for each config.</li>
<li><code class="highlighter-rouge">pool:</code> For test we use Ecto.Adapters.SQL.Sandbox. This uses a special Sandbox pool that allows us to use test transactions.</li>
</ul>
<p>For Ecto testing we want to run each test in its own database transaction. We can do this by adding the following to test/test_helper.exs:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Mix</span><span class="o">.</span><span class="no">Task</span><span class="o">.</span><span class="n">run</span> <span class="sd">"</span><span class="s2">ecto.create"</span><span class="p">,</span> <span class="p">[</span><span class="sd">"</span><span class="s2">--quiet"</span><span class="p">]</span>
<span class="no">Mix</span><span class="o">.</span><span class="no">Task</span><span class="o">.</span><span class="n">run</span> <span class="sd">"</span><span class="s2">ecto.migrate"</span><span class="p">,</span> <span class="p">[</span><span class="sd">"</span><span class="s2">--quiet"</span><span class="p">]</span>
<span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">SQL</span><span class="o">.</span><span class="n">begin_test_transaction</span><span class="p">(</span><span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="p">)</span></code></pre></figure>
<p>At this point the initial test runs which means our config is good:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="gp">$ </span>mix <span class="nb">test
</span>14:58:37.285 <span class="o">[</span>debug] BEGIN <span class="o">[]</span> OK <span class="nv">query</span><span class="o">=</span>0.3ms
14:58:37.286 <span class="o">[</span>debug] SAVEPOINT ecto_sandbox <span class="o">[]</span> OK <span class="nv">query</span><span class="o">=</span>0.3ms
.
Finished <span class="k">in </span>0.1 seconds <span class="o">(</span>0.1s on load, 0.00s on tests<span class="o">)</span>
1 <span class="nb">test</span>, 0 failures
Randomized with seed 323271</code></pre></figure>
<p>Now that we have Ecto setup for our tests we can write a real test.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">we can create and insert object records"</span> <span class="k">do</span>
<span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">insert!</span> <span class="p">%</span><span class="no">Ee</span><span class="o">.</span><span class="no">Object</span><span class="p">{</span>
<span class="ss">title:</span> <span class="sd">"</span><span class="s2">Test Object"</span><span class="p">,</span>
<span class="ss">description:</span> <span class="sd">"</span><span class="s2">And object for the test"</span>
<span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>And this simple test passes.</p>
<p>I’ll commit and add a tag called experiment/simple_test.</p>
<h3 id="experiment-3---simple-query">Experiment 3 - Simple Query</h3>
<p>In this experiment we’ll add a simple query and wrap it in a function. I’ll start by writing a test that defines the interface I want:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">we can insert object records and query them back"</span> <span class="k">do</span>
<span class="n">title</span> <span class="o">=</span> <span class="sd">"</span><span class="s2">Queryable title"</span>
<span class="n">object</span> <span class="o">=</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">insert!</span> <span class="p">%</span><span class="no">Ee</span><span class="o">.</span><span class="no">Object</span><span class="p">{</span>
<span class="ss">title:</span> <span class="n">title</span><span class="p">,</span>
<span class="ss">description:</span> <span class="sd">"</span><span class="s2">And object for the test"</span>
<span class="p">}</span>
<span class="n">found</span> <span class="o">=</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Object</span><span class="o">.</span><span class="n">find_by_title</span><span class="p">(</span><span class="n">title</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">all</span>
<span class="n">assert</span> <span class="no">Enum</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="n">found</span><span class="p">)</span> <span class="o">==</span> <span class="m">1</span>
<span class="n">assert</span> <span class="no">List</span><span class="o">.</span><span class="n">first</span><span class="p">(</span><span class="n">found</span><span class="p">)</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">object</span><span class="o">.</span><span class="n">id</span>
<span class="k">end</span></code></pre></figure>
<p>This test</p>
<ol>
<li>Inserts an object with a special title</li>
<li>Uses <code class="highlighter-rouge">Ee.Object.find_by_title/1</code> to query the same object.</li>
<li>Asserts that it got the object it expected.</li>
</ol>
<p>To make this test pass we just need to implement <code class="highlighter-rouge">find_by_title/1</code>. The query is simply:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">find_by_title</span><span class="p">(</span><span class="n">title</span><span class="p">)</span> <span class="k">do</span>
<span class="n">from</span> <span class="n">object</span> <span class="ow">in</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Object</span><span class="p">,</span> <span class="ss">where:</span> <span class="n">object</span><span class="o">.</span><span class="n">title</span> <span class="o">==</span> <span class="o">^</span><span class="n">title</span>
<span class="k">end</span></code></pre></figure>
<p>We query from all Ee.Object records and select those where the title matches our specified <code class="highlighter-rouge">title</code>. With this implementation the test passes.</p>
<p>There are no surprises here but we’re building slowly toward our goal.</p>
<p>I’ll commit and add a tag experiment/simple_query.</p>
<h3 id="experiment-4---datetime-query">Experiment 4 - DateTime Query</h3>
<p>Let’s take another step toward our goal and introduce a DateTime field and query. We’ll continue to take things slow and will test out using a DateTime field in <code class="highlighter-rouge">Object</code> before trying an association.</p>
<p>To add the field we’ll need a new migration:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">AddDateTimeToObject</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="k">def</span> <span class="n">change</span> <span class="k">do</span>
<span class="n">alter</span> <span class="n">table</span><span class="p">(</span><span class="ss">:object</span><span class="p">)</span> <span class="k">do</span>
<span class="n">add</span> <span class="ss">:date</span><span class="p">,</span> <span class="ss">:datetime</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>and to update the <code class="highlighter-rouge">Object</code> struct:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gd">--- a/lib/ee/object.ex
</span><span class="gi">+++ b/lib/ee/object.ex
</span><span class="gu">@@ -5,6 +5,7 @@ defmodule Ee.Object do
</span> schema "object" do
field :title, :string
field :description, :string
<span class="gi">+ field :date, Ecto.DateTime
</span>
<span class="err"> # has_many :available_date_ranges, Project.AvailableDateRange, on_replace: :delete</span></code></pre></figure>
<p>With this let’s try inserting an object with a filled in date.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">we can create and insert object records with the date field"</span> <span class="k">do</span>
<span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">insert!</span> <span class="p">%</span><span class="no">Ee</span><span class="o">.</span><span class="no">Object</span><span class="p">{</span>
<span class="ss">title:</span> <span class="sd">"</span><span class="s2">Test Object"</span><span class="p">,</span>
<span class="ss">description:</span> <span class="sd">"</span><span class="s2">And object for the test"</span><span class="p">,</span>
<span class="ss">date:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">DateTime</span><span class="o">.</span><span class="n">from_erl</span><span class="p">({</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">1</span><span class="p">,</span> <span class="m">6</span><span class="p">},</span> <span class="p">{</span><span class="m">3</span><span class="p">,</span> <span class="m">59</span><span class="p">,</span> <span class="m">5</span><span class="p">}})</span>
<span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>And this test passes.</p>
<p>Now, let’s try to query the object by date. Here’s the test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">we can insert object records and query them back by date"</span> <span class="k">do</span>
<span class="n">date</span> <span class="o">=</span> <span class="p">{</span> <span class="p">{</span><span class="m">2016</span><span class="p">,</span> <span class="m">1</span><span class="p">,</span> <span class="m">6</span><span class="p">},</span> <span class="p">{</span><span class="m">8</span><span class="p">,</span> <span class="m">0</span><span class="p">,</span> <span class="m">0</span><span class="p">}}</span>
<span class="n">object</span> <span class="o">=</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">insert!</span> <span class="p">%</span><span class="no">Ee</span><span class="o">.</span><span class="no">Object</span><span class="p">{</span>
<span class="ss">title:</span> <span class="sd">"</span><span class="s2">Test Object"</span><span class="p">,</span>
<span class="ss">description:</span> <span class="sd">"</span><span class="s2">And object for the test"</span><span class="p">,</span>
<span class="ss">date:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">DateTime</span><span class="o">.</span><span class="n">from_erl</span><span class="p">(</span><span class="n">date</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">found</span> <span class="o">=</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Object</span><span class="o">.</span><span class="n">find_by_date</span><span class="p">(</span><span class="n">date</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">all</span>
<span class="n">assert</span> <span class="no">Enum</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="n">found</span><span class="p">)</span> <span class="o">==</span> <span class="m">1</span>
<span class="n">assert</span> <span class="no">List</span><span class="o">.</span><span class="n">first</span><span class="p">(</span><span class="n">found</span><span class="p">)</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">object</span><span class="o">.</span><span class="n">id</span>
<span class="k">end</span></code></pre></figure>
<p>And here’s a working implementation of find_by_date:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">find_by_date</span><span class="p">(</span><span class="n">date</span><span class="p">)</span> <span class="k">do</span>
<span class="n">from</span> <span class="n">object</span> <span class="ow">in</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Object</span><span class="p">,</span> <span class="ss">where:</span> <span class="n">object</span><span class="o">.</span><span class="n">date</span> <span class="o">==</span> <span class="o">^</span><span class="n">date</span>
<span class="k">end</span></code></pre></figure>
<p>This test passes and the experiment is a success. I commit and add a tag experiment/date-query.</p>
<h3 id="experiment-4---timex-query">Experiment 4 - Timex Query</h3>
<p>Now, I don’t actually want to use <code class="highlighter-rouge">Ecto.DateTime</code> to store dates. I want to use the <a href="https://github.com/bitwalker/timex">timex</a> and <a href="https://github.com/bitwalker/timex_ecto">timex_ecto</a> libraries for their improved handling of timezones.</p>
<p>Let’s start by setting up Timex in our application. I’ll follow the <a href="https://github.com/bitwalker/timex_ecto/blob/master/README.md">README</a>.</p>
<p>First I’ll update mix.exs to include the timex and timex_ecto dependencies and start the tzdata app:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gd">--- a/mix.exs
</span><span class="gi">+++ b/mix.exs
</span><span class="gu">@@ -14,7 +14,7 @@ defmodule Ee.Mixfile do
</span> #
# Type "mix help compile.app" for more information
def application do
<span class="gd">- [applications: [:logger, :postgrex, :ecto],
</span><span class="gi">+ [applications: [:logger, :postgrex, :ecto, :tzdata, :timex],
</span> mod: {Ee, []}]
end
<span class="gu">@@ -31,6 +31,8 @@ defmodule Ee.Mixfile do
</span> [
{:postgrex, ">= 0.0.0"},
{:ecto, "~> 1.0"},
<span class="gi">+ {:timex, "~> 1.0"},
+ {:timex_ecto, "~> 0.7"}
</span> ]
end
<span class="err"> end</span></code></pre></figure>
<p>I want to use the <code class="highlighter-rouge">Timex.Ecto.DateTimeWithTimezone</code> type which is only supported with Postgres and requires some extra setup. We have to run some custom SQL which we can do in a migration like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">AddTimexDatetimetz</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="k">def</span> <span class="n">up</span> <span class="k">do</span>
<span class="n">execute</span> <span class="sd">"""
CREATE TYPE datetimetz AS (
dt timestamptz,
tz varchar
);
"""</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">down</span> <span class="k">do</span>
<span class="n">execute</span> <span class="sd">"</span><span class="s2">DROP TYPE IF EXISTS datetimetz;"</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>When bringing up this migration we execute custom SQL to create a new type called “datetimetz”. This SQL comes from the <a href="https://timex.readme.io/v0.19/docs/using-with-ecto#section-using-datetimewithtimezone">timex_ecto documentation</a>.</p>
<p>Now that timex is setup we can add a field to <code class="highlighter-rouge">Object</code>. I’ve simply done this:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gd">--- a/lib/ee/object.ex
</span><span class="gi">+++ b/lib/ee/object.ex
</span><span class="gu">@@ -6,6 +6,7 @@ defmodule Ee.Object do
</span> field :title, :string
field :description, :string
field :date, Ecto.DateTime
<span class="gi">+ field :timex, Timex.Ecto.DateTimeWithTimezone
</span>
<span class="err"> # has_many :available_date_ranges, Project.AvailableDateRange, on_replace: :delete</span></code></pre></figure>
<p>I’ve named the field <code class="highlighter-rouge">timex</code> so that it is clear which kind of date field it is in these examples.</p>
<p>And of course to add this field we need another migration:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Ee</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">AddTimexToObject</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="k">def</span> <span class="n">change</span> <span class="k">do</span>
<span class="n">alter</span> <span class="n">table</span><span class="p">(</span><span class="ss">:object</span><span class="p">)</span> <span class="k">do</span>
<span class="n">add</span> <span class="ss">:timex</span><span class="p">,</span> <span class="ss">:datetimetz</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>At this point my tests start to fail.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix test
11:04:58.985 [debug] BEGIN [] OK query=0.2ms
11:04:58.985 [debug] SAVEPOINT ecto_sandbox [] OK query=0.2ms
.
11:04:59.063 [debug] INSERT INTO "object" ("inserted_at", "updated_at", "date", "description", "timex", "title") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id" [{ {2016, 1, 6}, {19, 4, 59, 0}}, { {2016, 1, 6}, {19, 4, 59, 0}}, { {2016, 1, 6}, {8, 0, 0, 0}}, "And object for the test", nil, "Test Object"] OK query=0.9ms
11:04:59.074 [debug] INSERT INTO "object" ("inserted_at", "updated_at", "date", "description", "timex", "title") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id" [{ {2016, 1, 6}, {19, 4, 59, 0}}, { {2016, 1, 6}, {19, 4, 59, 0}}, { {2016, 1, 6}, {3, 59, 5, 0}}, "And object for the test", nil, "Test Object"] OK query=5.0ms
11:04:59.076 [debug] INSERT INTO "object" ("inserted_at", "updated_at", "date", "description", "timex", "title") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id" [{ {2016, 1, 6}, {19, 4, 59, 0}}, { {2016, 1, 6}, {19, 4, 59, 0}}, nil, "And object for the test", nil, "Queryable title"] OK query=1.0ms
1) test we can insert object records and query them back by date (EeTest)
test/ee_test.exs:39
** (CaseClauseError) no case clause matching: {:ok, {:error, %ArgumentError{message: "no extension found for oid `131165`"}, [{Postgrex.Types, :fetch!, 2, [file: 'lib/postgrex/types.ex', line: 296]}, {Postgrex.Types, :decoder, 2, [file: 'lib/postgrex/types.ex', line: 220]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1088]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1088]}, {Postgrex.Protocol, :decoders, 2, [file: 'lib/postgrex/protocol.ex', line: 545]}, {Postgrex.Protocol, :describe_fields, 4, [file: 'lib/postgrex/protocol.ex', line: 368]}, {Postgrex.Protocol, :query, 4, [file: 'lib/postgrex/protocol.ex', line: 80]}, {Postgrex.Connection, :query, 6, [file: 'lib/postgrex/connection.ex', line: 315]}, {Postgrex.Connection, :query, 4, [file: 'lib/postgrex/connection.ex', line: 101]}, {Ecto.Adapters.Postgres.Connection, :query, 4, [file: 'lib/ecto/adapters/postgres/connection.ex', line: 31]}, {:timer, :tc, 3, [file: 'timer.erl', line: 197]}, {Ecto.Adapters.SQL, :query, 7, [file: 'lib/ecto/adapters/sql.ex', line: 262]}, {Ecto.Pool, :do_run, 4, [file: 'lib/ecto/pool.ex', line: 159]}, {Ecto.Adapters.SQL, :query, 6, [file: 'lib/ecto/adapters/sql.ex', line: 247]}, {Ecto.Adapters.SQL, :query, 5, [file: 'lib/ecto/adapters/sql.ex', line: 225]}, {Ecto.Adapters.SQL, :query!, 5, [file: 'lib/ecto/adapters/sql.ex', line: 183]}, {Ecto.Adapters.SQL, :execute, 6, [file: 'lib/ecto/adapters/sql.ex', line: 481]}, {Ecto.Repo.Queryable, :execute, 5, [file: 'lib/ecto/repo/queryable.ex', line: 95]}, {Ecto.Repo.Queryable, :all, 4, [file: 'lib/ecto/repo/queryable.ex', line: 15]}, {EeTest, :"test we can insert object records and query them back by date", 1, [file: 'test/ee_test.exs', line: 49]}]}, %{}, [], ""}
stacktrace:
(postgrex) lib/postgrex/protocol.ex:80: Postgrex.Protocol.query/4
(postgrex) lib/postgrex/connection.ex:315: Postgrex.Connection.query/6
(postgrex) lib/postgrex/connection.ex:101: Postgrex.Connection.query/4
(ecto) lib/ecto/adapters/postgres/connection.ex:31: Ecto.Adapters.Postgres.Connection.query/4
(stdlib) timer.erl:197: :timer.tc/3
(ecto) lib/ecto/adapters/sql.ex:262: Ecto.Adapters.SQL.query/7
(ecto) lib/ecto/pool.ex:159: Ecto.Pool.do_run/4
(ecto) lib/ecto/adapters/sql.ex:247: Ecto.Adapters.SQL.query/6
(ecto) lib/ecto/adapters/sql.ex:225: Ecto.Adapters.SQL.query/5
(ecto) lib/ecto/adapters/sql.ex:183: Ecto.Adapters.SQL.query!/5
(ecto) lib/ecto/adapters/sql.ex:481: Ecto.Adapters.SQL.execute/6
(ecto) lib/ecto/repo/queryable.ex:95: Ecto.Repo.Queryable.execute/5
(ecto) lib/ecto/repo/queryable.ex:15: Ecto.Repo.Queryable.all/4
test/ee_test.exs:49
.
2) test we can insert object records and query them back (EeTest)
test/ee_test.exs:16
** (CaseClauseError) no case clause matching: {:ok, {:error, %ArgumentError{message: "no extension found for oid `131165`"}, [{Postgrex.Types, :fetch!, 2, [file: 'lib/postgrex/types.ex', line: 296]}, {Postgrex.Types, :decoder, 2, [file: 'lib/postgrex/types.ex', line: 220]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1088]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1088]}, {Postgrex.Protocol, :decoders, 2, [file: 'lib/postgrex/protocol.ex', line: 545]}, {Postgrex.Protocol, :describe_fields, 4, [file: 'lib/postgrex/protocol.ex', line: 368]}, {Postgrex.Protocol, :query, 4, [file: 'lib/postgrex/protocol.ex', line: 80]}, {Postgrex.Connection, :query, 6, [file: 'lib/postgrex/connection.ex', line: 315]}, {Postgrex.Connection, :query, 4, [file: 'lib/postgrex/connection.ex', line: 101]}, {Ecto.Adapters.Postgres.Connection, :query, 4, [file: 'lib/ecto/adapters/postgres/connection.ex', line: 31]}, {:timer, :tc, 3, [file: 'timer.erl', line: 197]}, {Ecto.Adapters.SQL, :query, 7, [file: 'lib/ecto/adapters/sql.ex', line: 262]}, {Ecto.Pool, :do_run, 4, [file: 'lib/ecto/pool.ex', line: 159]}, {Ecto.Adapters.SQL, :query, 6, [file: 'lib/ecto/adapters/sql.ex', line: 247]}, {Ecto.Adapters.SQL, :query, 5, [file: 'lib/ecto/adapters/sql.ex', line: 225]}, {Ecto.Adapters.SQL, :query!, 5, [file: 'lib/ecto/adapters/sql.ex', line: 183]}, {Ecto.Adapters.SQL, :execute, 6, [file: 'lib/ecto/adapters/sql.ex', line: 481]}, {Ecto.Repo.Queryable, :execute, 5, [file: 'lib/ecto/repo/queryable.ex', line: 95]}, {Ecto.Repo.Queryable, :all, 4, [file: 'lib/ecto/repo/queryable.ex', line: 15]}, {EeTest, :"test we can insert object records and query them back", 1, [file: 'test/ee_test.exs', line: 25]}]}, %{}, [], ""}
stacktrace:
(postgrex) lib/postgrex/protocol.ex:80: Postgrex.Protocol.query/4
(postgrex) lib/postgrex/connection.ex:315: Postgrex.Connection.query/6
(postgrex) lib/postgrex/connection.ex:101: Postgrex.Connection.query/4
(ecto) lib/ecto/adapters/postgres/connection.ex:31: Ecto.Adapters.Postgres.Connection.query/4
(stdlib) timer.erl:197: :timer.tc/3
(ecto) lib/ecto/adapters/sql.ex:262: Ecto.Adapters.SQL.query/7
(ecto) lib/ecto/pool.ex:159: Ecto.Pool.do_run/4
(ecto) lib/ecto/adapters/sql.ex:247: Ecto.Adapters.SQL.query/6
(ecto) lib/ecto/adapters/sql.ex:225: Ecto.Adapters.SQL.query/5
(ecto) lib/ecto/adapters/sql.ex:183: Ecto.Adapters.SQL.query!/5
(ecto) lib/ecto/adapters/sql.ex:481: Ecto.Adapters.SQL.execute/6
(ecto) lib/ecto/repo/queryable.ex:95: Ecto.Repo.Queryable.execute/5
(ecto) lib/ecto/repo/queryable.ex:15: Ecto.Repo.Queryable.all/4
test/ee_test.exs:25
3) test we can create and insert object records (EeTest)
test/ee_test.exs:9
** (ArgumentError) repo Ee.Repo is not started, please ensure it is part of your supervision tree
stacktrace:
(ecto) lib/ecto/adapters/sql.ex:253: Ecto.Adapters.SQL.query/6
(ecto) lib/ecto/adapters/sql.ex:225: Ecto.Adapters.SQL.query/5
(ecto) lib/ecto/adapters/sql.ex:487: Ecto.Adapters.SQL.model/6
(ecto) lib/ecto/repo/schema.ex:297: Ecto.Repo.Schema.apply/5
(ecto) lib/ecto/repo/schema.ex:81: anonymous fn/11 in Ecto.Repo.Schema.do_insert/4
(ecto) lib/ecto/repo/schema.ex:14: Ecto.Repo.Schema.insert!/4
test/ee_test.exs:10
Finished in 0.5 seconds (0.4s on load, 0.02s on tests)
5 tests, 3 failures
Randomized with seed 53554
</code></pre>
</div>
<p>And they fail intermittently. Actually, I discovered that they fail the first time I run them and then they don’t fail again until I drop the database with:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="gp">$ </span><span class="nv">MIX_ENV</span><span class="o">=</span><span class="nb">test </span>mix ecto.drop</code></pre></figure>
<p>I can make my results more consistent by making this change to the test helper:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gd">--- a/test/test_helper.exs
</span><span class="gi">+++ b/test/test_helper.exs
</span><span class="gu">@@ -1,5 +1,6 @@
</span> ExUnit.start()
<span class="gi">+Mix.Task.run "ecto.drop", ["--quiet"]
</span> Mix.Task.run "ecto.create", ["--quiet"]
Mix.Task.run "ecto.migrate", ["--quiet"]
<span class="err"> Ecto.Adapters.SQL.begin_test_transaction(Ee.Repo)</span></code></pre></figure>
<p>This gives me a failure every time I run the tests but the number of failures varies. To try to understand what’s going wrong I comment out all of the tests except the initial test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">the truth"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="m">1</span> <span class="o">+</span> <span class="m">1</span> <span class="o">==</span> <span class="m">2</span>
<span class="k">end</span></code></pre></figure>
<p>This passes every time I run it. Continuing with baby steps, I uncomment this test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">we can create and insert object records"</span> <span class="k">do</span></code></pre></figure>
<p>The two uncommented tests pass every time I run them. Next, I uncomment another insertion test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">we can create and insert object records with the date field"</span> <span class="k">do</span></code></pre></figure>
<p>The three tests pass every time I run them. Next, I uncomment the first query test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">we can insert object records and query them back"</span> <span class="k">do</span></code></pre></figure>
<p>And this fails. So there is some problem with query?</p>
<h3 id="experiment-5---query-tests">Experiment 5 - Query Tests</h3>
<p>To isolate the problem, I comment out all the tests except “we can insert object records and query them back” and this single test fails every time I run it:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix test
11:13:18.923 [debug] BEGIN [] OK query=0.2ms
11:13:18.924 [debug] SAVEPOINT ecto_sandbox [] OK query=0.4ms
11:13:18.984 [debug] INSERT INTO "object" ("inserted_at", "updated_at", "date", "description", "timex", "title") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id" [{ {2016, 1, 6}, {19, 13, 18, 0}}, { {2016, 1, 6}, {19, 13, 18, 0}}, nil, "And object for the test", nil, "Queryable title"] OK query=0.9ms
1) test we can insert object records and query them back (EeTest)
test/ee_test.exs:16
** (CaseClauseError) no case clause matching: {:ok, {:error, %ArgumentError{message: "no extension found for oid `131645`"}, [{Postgrex.Types, :fetch!, 2, [file: 'lib/postgrex/types.ex', line: 296]}, {Postgrex.Types, :decoder, 2, [file: 'lib/postgrex/types.ex', line: 220]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1088]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1088]}, {Postgrex.Protocol, :decoders, 2, [file: 'lib/postgrex/protocol.ex', line: 545]}, {Postgrex.Protocol, :describe_fields, 4, [file: 'lib/postgrex/protocol.ex', line: 368]}, {Postgrex.Protocol, :query, 4, [file: 'lib/postgrex/protocol.ex', line: 80]}, {Postgrex.Connection, :query, 6, [file: 'lib/postgrex/connection.ex', line: 315]}, {Postgrex.Connection, :query, 4, [file: 'lib/postgrex/connection.ex', line: 101]}, {Ecto.Adapters.Postgres.Connection, :query, 4, [file: 'lib/ecto/adapters/postgres/connection.ex', line: 31]}, {:timer, :tc, 3, [file: 'timer.erl', line: 197]}, {Ecto.Adapters.SQL, :query, 7, [file: 'lib/ecto/adapters/sql.ex', line: 262]}, {Ecto.Pool, :do_run, 4, [file: 'lib/ecto/pool.ex', line: 159]}, {Ecto.Adapters.SQL, :query, 6, [file: 'lib/ecto/adapters/sql.ex', line: 247]}, {Ecto.Adapters.SQL, :query, 5, [file: 'lib/ecto/adapters/sql.ex', line: 225]}, {Ecto.Adapters.SQL, :query!, 5, [file: 'lib/ecto/adapters/sql.ex', line: 183]}, {Ecto.Adapters.SQL, :execute, 6, [file: 'lib/ecto/adapters/sql.ex', line: 481]}, {Ecto.Repo.Queryable, :execute, 5, [file: 'lib/ecto/repo/queryable.ex', line: 95]}, {Ecto.Repo.Queryable, :all, 4, [file: 'lib/ecto/repo/queryable.ex', line: 15]}, {EeTest, :"test we can insert object records and query them back", 1, [file: 'test/ee_test.exs', line: 25]}]}, %{}, [], ""}
stacktrace:
(postgrex) lib/postgrex/protocol.ex:80: Postgrex.Protocol.query/4
(postgrex) lib/postgrex/connection.ex:315: Postgrex.Connection.query/6
(postgrex) lib/postgrex/connection.ex:101: Postgrex.Connection.query/4
(ecto) lib/ecto/adapters/postgres/connection.ex:31: Ecto.Adapters.Postgres.Connection.query/4
(stdlib) timer.erl:197: :timer.tc/3
(ecto) lib/ecto/adapters/sql.ex:262: Ecto.Adapters.SQL.query/7
(ecto) lib/ecto/pool.ex:159: Ecto.Pool.do_run/4
(ecto) lib/ecto/adapters/sql.ex:247: Ecto.Adapters.SQL.query/6
(ecto) lib/ecto/adapters/sql.ex:225: Ecto.Adapters.SQL.query/5
(ecto) lib/ecto/adapters/sql.ex:183: Ecto.Adapters.SQL.query!/5
(ecto) lib/ecto/adapters/sql.ex:481: Ecto.Adapters.SQL.execute/6
(ecto) lib/ecto/repo/queryable.ex:95: Ecto.Repo.Queryable.execute/5
(ecto) lib/ecto/repo/queryable.ex:15: Ecto.Repo.Queryable.all/4
test/ee_test.exs:25
</code></pre>
</div>
<p>What does this error mean? It sounds like some extension isn’t enabled in the database but which extension?</p>
<p>Debugging this head on hasn’t been fruitful. The oid 131645 doesn’t seem to have a specific meaning that I can find. It even seems to change every time I run the tests.</p>
<p>I’m going to try a less direct approach. First, I remove the <code class="highlighter-rouge">mix ecto.drop</code> call that I added to test_helper. Then I run the test again and see it pass. This suggests that there is some data being carried over in the database that is making the test pass! I should be able to look in the database to find it.</p>
<p>I”m using PG Commander to look at the database.</p>
<ul>
<li>There are no objects left in the <code class="highlighter-rouge">object</code> table.</li>
<li>There is a datetimetz user defined type.</li>
</ul>
<p>I’m sort of thinking that the datetimetz type isn’t getting setup when the tests are running.</p>
<h3 id="experiment-6---run-commands-from-the-shell">Experiment 6 - Run commands from the shell</h3>
<p>At this point we’re into real investigation and it’s time to get a little more scientific about these experiments. Based on my thoughts above, I have a hypothesis: Something isn’t working right with the <code class="highlighter-rouge">Mix.Task.run</code> calls. Perhaps they don’t finish by the time the test starts.</p>
<p>Based on this hypothesis I ran the following experiment.</p>
<p>First, I commented out all the <code class="highlighter-rouge">Mix.Task</code> calls from the test helper:</p>
<figure class="highlight"><pre><code class="language-diff" data-lang="diff"><span class="gd">--- a/test/test_helper.exs
</span><span class="gi">+++ b/test/test_helper.exs
</span><span class="gu">@@ -1,6 +1,6 @@
</span> ExUnit.start()
<span class="gd">-Mix.Task.run "ecto.drop", ["--quiet"]
-Mix.Task.run "ecto.create", ["--quiet"]
-Mix.Task.run "ecto.migrate", ["--quiet"]
</span><span class="gi">+# Mix.Task.run "ecto.drop", ["--quiet"]
+# Mix.Task.run "ecto.create", ["--quiet"]
+# Mix.Task.run "ecto.migrate", ["--quiet"]
</span><span class="err"> Ecto.Adapters.SQL.begin_test_transaction(Ee.Repo)</span></code></pre></figure>
<p>Second, I ran this sequence of shell commands</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix ecto.drop --quiet ; mix ecto.create --quiet ; mix ecto.migrate --quiet ; mix test
</code></pre>
</div>
<p>This sequence of commands passes consistently. So I think this points to some problem with the commands running properly.</p>
<h2 id="next-steps">Next steps</h2>
<p>Unfortunately, this post is getting a little too long so I am going to have to stop here and continue this investigation in a second post.</p>
<p>In this post we reviewed setting up Ecto and then worked through a system of experiments in order to driver and investigation of a problem that I’m having in another project. I hope you enjoyed exploring this system of problem isolation with me.</p>
<p>Next week I’ll continue in this line of investigation and will pick up where we just left off.</p>
<p><a href="http://learningelixir.joekain.com/experiments-with-ecto-queries/">Read more...</a></p>
http://learningelixir.joekain.com/setting-up-ecto-in-elixir2015-12-15T00:00:00-08:002015-12-15T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>It’s been a few weeks since I last wrote about <a href="/building-a-cache-in-elixir-with-ets/">Domain Scrapper</a>. I’ve covered fetching data from services like Reddit and Twitter and for a few posts I’ve been promising to start looking at aggregation of the collected data. In this post I want to start on that task.</p>
<h2 id="ecto">Ecto</h2>
<p>As I’m sure you’ve heard already, Ecto is the database library to use with Elixir. I’m going to set it up within Domain Scrapper.</p>
<p>I’ve used Ecto in a number of Phoenix based projects before and Phoenix graciously sets up Ecto as part of its initial project. But, recently I wrote a simple script to fetch Elixir github repos as part of my <a href="/idiomatic-elixir/"><em>Idiomatic Elixir</em></a> research. This was the first time I had to set up Ecto by hand and it was a learning experience.</p>
<p>In this post I’m going to repeat the process with Domain Scrapper.</p>
<h3 id="create-a-new-aggregator-application-under-the-umbrella">Create a New Aggregator Application Under the Umbrella</h3>
<p>Domain Scrapper is an umbrella application and aggregation will be
handled in a new application. So first we’ll create the aggregator:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix new aggregator
* creating README.md
* creating .gitignore
* creating mix.exs
* creating config
* creating config/config.exs
* creating lib
* creating lib/aggregator.ex
* creating test
* creating test/test_helper.exs
* creating test/aggregator_test.exs
Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:
cd aggregator
mix test
Run "mix help" for more commands.
</code></pre>
</div>
<p>With this, we can begin to setup and configure Ecto within the aggregator application.</p>
<h3 id="install-and-start-ecto">Install and Start Ecto</h3>
<p>We’ll be using Ecto with postgres so I’ll use the postgrex adapter.
To set this up we have to install Ecto and its dependencies by adding
these deps to apps/aggregator/mix.exs:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defp</span> <span class="n">deps</span> <span class="k">do</span>
<span class="p">[</span>
<span class="p">{</span><span class="ss">:postgrex</span><span class="p">,</span> <span class="sd">"</span><span class="s2">>= 0.0.0"</span><span class="p">},</span>
<span class="p">{</span><span class="ss">:ecto</span><span class="p">,</span> <span class="sd">"</span><span class="s2">~> 1.0"</span><span class="p">},</span>
<span class="p">]</span>
<span class="k">end</span></code></pre></figure>
<p>Also, we have to start both the Ecto and postgrex applications:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">application</span> <span class="k">do</span>
<span class="p">[</span><span class="ss">applications:</span> <span class="p">[</span><span class="ss">:logger</span><span class="p">,</span> <span class="ss">:postgrex</span><span class="p">,</span> <span class="ss">:ecto</span><span class="p">]]</span>
<span class="k">end</span></code></pre></figure>
<p>Then run <code class="highlighter-rouge">mix deps.get</code> from the project root.</p>
<h3 id="configure-the-database">Configure the Database</h3>
<p>The next step is to configure the database in apps/aggregator/config.exs.</p>
<p>I’ll start with the development configuration. At some point I’ll need to come back to this and add a production configuration.</p>
<p>I’ve added the following to config.exs:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">config</span> <span class="ss">:aggregator</span><span class="p">,</span> <span class="no">Aggregator</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">adapter:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">Postgres</span><span class="p">,</span>
<span class="ss">database:</span> <span class="sd">"</span><span class="s2">domain_scrapper"</span><span class="p">,</span>
<span class="ss">username:</span> <span class="sd">"</span><span class="s2">postgres"</span><span class="p">,</span>
<span class="ss">password:</span> <span class="sd">"</span><span class="s2">postgres"</span></code></pre></figure>
<p>You will need to use a valid username / password for your postgres installation.</p>
<h3 id="write-the-repo-module">Write the Repo Module</h3>
<p>Next, we have to create a <code class="highlighter-rouge">Repo</code> module which describes the OTP
application in use. I believe this is used to find our configuration
above:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Aggregator</span><span class="o">.</span><span class="no">Repo</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">otp_app:</span> <span class="ss">:aggregator</span>
<span class="k">end</span></code></pre></figure>
<p>We can now run</p>
<div class="highlighter-rouge"><pre class="highlight"><code>mix ecto.create -r Aggregator.Repo
</code></pre>
</div>
<p>from the root of the project. This will will create the database. The “-r” option was necessary to specify which Repo to create. From an umbrella app we could easily end up with multiple repos across a number of applications. Though for Domain Scrapper this is our first.</p>
<p>I verified the database was created using PG Commander to see the newly created database.</p>
<h3 id="create-a-model-and-a-migration">Create a Model and a Migration</h3>
<p>Next, we need to create a model but I haven’t fully thought through what’s going into the aggregator yet. I know at the very least that we have a URL to save. So, let’s start with that much by creating apps/aggregator/lib/aggregator/domain.ex:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Aggregator</span><span class="o">.</span><span class="no">Domain</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Schema</span>
<span class="n">schema</span> <span class="sd">"</span><span class="s2">domain"</span> <span class="k">do</span>
<span class="n">field</span> <span class="ss">:url</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>We use <code class="highlighter-rouge">Ecto.Schema</code> to pull in all the Ecto code for defining a model. Then we define our “domain” schema with a single <code class="highlighter-rouge">url</code> field.</p>
<p>Now, before we can use this we need to write and run a migration to build the associated table in the database.</p>
<p>We can use the Ecto’s migration generator to help us get started. I ran the following from within the aggregator application:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix ecto.gen.migration add_domain
* creating priv/repo/migrations
* creating priv/repo/migrations/20151215005708_add_domain.exs
</code></pre>
</div>
<p>This gives us an empty migration in priv/repo/migrations/20151215005708_add_domain.exs. Of course your filename may differer somewhat. We start with:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Aggregator</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">AddDomain</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="k">def</span> <span class="n">change</span> <span class="k">do</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>We need to fill this in so that it contains:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Aggregator</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">AddDomain</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="k">def</span> <span class="n">change</span> <span class="k">do</span>
<span class="n">create</span> <span class="n">table</span><span class="p">(</span><span class="ss">:domain</span><span class="p">)</span> <span class="k">do</span>
<span class="n">add</span> <span class="ss">:url</span><span class="p">,</span> <span class="ss">:string</span>
<span class="n">timestamps</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>This creates the table and configures the columns to match our schema. With this we can run the migration to update the database:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix ecto.migrate
06:33:47.015 [info] == Running Aggregator.Repo.Migrations.AddDomain.change/0 forward
06:33:47.015 [info] create table domain
06:33:47.024 [info] == Migrated in 0.0s
</code></pre>
</div>
<p>I can use PG Commander again to verify the new table looks as I expect.</p>
<h3 id="use-iex-to-build-a-model">Use iex to Build a Model</h3>
<p>With this let’s try out the model using iex</p>
<div class="highlighter-rouge"><pre class="highlight"><code>iex(1)> Aggregator.Repo.insert! %Aggregator.Domain{url: "http://about.joekain.com"}
** (ArgumentError) repo Aggregator.Repo is not started, please ensure it is part of your supervision tree
(ecto) lib/ecto/adapters/sql.ex:250: Ecto.Adapters.SQL.query/6
(ecto) lib/ecto/adapters/sql.ex:222: Ecto.Adapters.SQL.query/5
(ecto) lib/ecto/adapters/sql.ex:484: Ecto.Adapters.SQL.model/6
(ecto) lib/ecto/repo/model.ex:253: Ecto.Repo.Model.apply/4
(ecto) lib/ecto/repo/model.ex:83: anonymous fn/10 in Ecto.Repo.Model.do_insert/4
(ecto) lib/ecto/repo/model.ex:14: Ecto.Repo.Model.insert!/4
</code></pre>
</div>
<p>Well, that didn’t work.</p>
<p>But, Ecto is kind enough to tell us that we have to start the <code class="highlighter-rouge">Repo</code> and suggests we do this in our supervision tree. This should be easily done except that when I went to do it I realized that I had not created the aggregator application as supervised using <code class="highlighter-rouge">mix new --sup</code>.</p>
<h2 id="supervise-an-elixir-application-from-scratch">Supervise an Elixir Application from Scratch</h2>
<p>At first I thought to start over with a supervised application and go through all the steps again. But then I thought this would be a good learning experience to understand how to setup a supervised application by hand.</p>
<p>To figure out what I needed to do I created a dummy supervised app in my umbrella project:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ mix new --sup foo
* creating README.md
* creating .gitignore
* creating mix.exs
* creating config
* creating config/config.exs
* creating lib
* creating lib/foo.ex
* creating test
* creating test/test_helper.exs
* creating test/foo_test.exs
Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:
cd foo
mix test
Run "mix help" for more commands.
</code></pre>
</div>
<p>Looking at lib/foo.ex I see</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Foo</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Application</span>
<span class="c1"># See http://elixir-lang.org/docs/stable/elixir/Application.html</span>
<span class="c1"># for more information on OTP Applications</span>
<span class="k">def</span> <span class="n">start</span><span class="p">(</span><span class="n">_type</span><span class="p">,</span> <span class="n">_args</span><span class="p">)</span> <span class="k">do</span>
<span class="kn">import</span> <span class="no">Supervisor</span><span class="o">.</span><span class="no">Spec</span><span class="p">,</span> <span class="ss">warn:</span> <span class="no">false</span>
<span class="n">children</span> <span class="o">=</span> <span class="p">[</span>
<span class="c1"># Define workers and child supervisors to be supervised</span>
<span class="c1"># worker(Foo.Worker, [arg1, arg2, arg3]),</span>
<span class="p">]</span>
<span class="c1"># See http://elixir-lang.org/docs/stable/elixir/Supervisor.html</span>
<span class="c1"># for other strategies and supported options</span>
<span class="n">opts</span> <span class="o">=</span> <span class="p">[</span><span class="ss">strategy:</span> <span class="ss">:one_for_one</span><span class="p">,</span> <span class="ss">name:</span> <span class="no">Foo</span><span class="o">.</span><span class="no">Supervisor</span><span class="p">]</span>
<span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>contrast this with aggregator/lib/aggregator.ex:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Aggregator</span> <span class="k">do</span>
<span class="k">end</span></code></pre></figure>
<p>So, we need to <code class="highlighter-rouge">use Application</code> to pull in the <code class="highlighter-rouge">Application</code> boilerplate and then write a <code class="highlighter-rouge">start/2</code> function to start up the aggregator’s supervision tree. Something like this should work:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Aggregator</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Application</span>
<span class="k">def</span> <span class="n">start</span><span class="p">(</span><span class="n">_type</span><span class="p">,</span> <span class="n">_args</span><span class="p">)</span> <span class="k">do</span>
<span class="kn">import</span> <span class="no">Supervisor</span><span class="o">.</span><span class="no">Spec</span><span class="p">,</span> <span class="ss">warn:</span> <span class="no">false</span>
<span class="n">children</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">worker</span><span class="p">(</span><span class="no">Aggregator</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span> <span class="p">[])</span>
<span class="p">]</span>
<span class="n">opts</span> <span class="o">=</span> <span class="p">[</span><span class="ss">strategy:</span> <span class="ss">:one_for_one</span><span class="p">,</span> <span class="ss">name:</span> <span class="no">Aggregator</span><span class="o">.</span><span class="no">Supervisor</span><span class="p">]</span>
<span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Our <code class="highlighter-rouge">start/2</code> function defines a single child worker for <code class="highlighter-rouge">Aggregator.Repo</code> since this is the module we wanted to start up in the first place. Then it starts up a <code class="highlighter-rouge">Supervisor</code> with the same <code class="highlighter-rouge">:one_for_one</code> strategy that the mix would generate by default. Given that we have only one child the choice of strategy shouldn’t make much of a difference.</p>
<p>Continuing to follow the dummy app’s lead, we need to add the <code class="highlighter-rouge">Aggregator</code> module to the application definition in app/aggregator/mix.exs like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">application</span> <span class="k">do</span>
<span class="p">[</span><span class="ss">applications:</span> <span class="p">[</span><span class="ss">:logger</span><span class="p">,</span> <span class="ss">:postgrex</span><span class="p">,</span> <span class="ss">:ecto</span><span class="p">],</span>
<span class="ss">mod:</span> <span class="p">{</span><span class="no">Aggregator</span><span class="p">,</span> <span class="p">[]}]</span>
<span class="k">end</span></code></pre></figure>
<p>We added the <code class="highlighter-rouge">mod: {Aggregator, []}</code> element to describe the application’s module callback. This tells mix to use our <code class="highlighter-rouge">Aggregator.start/2</code> function to start up our application’s supervision tree.</p>
<p>Finally, we need to start the <code class="highlighter-rouge">:aggregator</code> application itself from the main application. We do this by adding it to the application list in apps/main/config.exs:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">application</span> <span class="k">do</span>
<span class="p">[</span><span class="ss">applications:</span> <span class="p">[</span><span class="ss">:logger</span><span class="p">,</span> <span class="ss">:unshortening_pool</span><span class="p">,</span> <span class="ss">:producer</span><span class="p">,</span> <span class="ss">:aggregator</span><span class="p">]]</span>
<span class="k">end</span></code></pre></figure>
<h2 id="use-iex-to-build-a-model-again">Use iex to Build a Model (again)</h2>
<p>With our supervision tree setup we again we try using iex to insert a record:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>iex(1)> Aggregator.Repo.insert! %Aggregator.Domain{url: "http://about.joekain.com"}
07:01:21.704 [debug] INSERT INTO "domain" ("inserted_at", "updated_at", "url") VALUES ($1, $2, $3) RETURNING "id" [nil, nil, "http://about.joekain.com"] ERROR query=86.9ms queue=4.9ms
** (Postgrex.Error) ERROR (not_null_violation): null value in column "inserted_at" violates not-null constraint
(ecto) lib/ecto/adapters/sql.ex:493: Ecto.Adapters.SQL.model/6
(ecto) lib/ecto/repo/model.ex:253: Ecto.Repo.Model.apply/4
(ecto) lib/ecto/repo/model.ex:83: anonymous fn/10 in Ecto.Repo.Model.do_insert/4
(ecto) lib/ecto/repo/model.ex:14: Ecto.Repo.Model.insert!/4
iex(1)>
07:01:25.020 [error] GenServer #PID<0.315.0> terminating
** (FunctionClauseError) no function clause matching in BlockingQueue.handle_call/3
lib/blocking_queue.ex:53: BlockingQueue.handle_call(:pop, {#PID<0.337.0>, #Reference<0.0.4.2>}, {1, [], :pop, {#PID<0.325.0>, #Reference<0.0.6.437>}})
(stdlib) gen_server.erl:629: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:661: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: :pop
State: {1, [], :pop, {#PID<0.325.0>, #Reference<0.0.6.437>}}
</code></pre>
</div>
<p>This gets a little further but fails to insert because the “inserted_at” column is null. It looks like we have to set this manually:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>iex(1)> Aggregator.Repo.insert! %Aggregator.Domain{url: "http://about.joekain.com", inserted_at: Ecto.DateTime.local, updated_at: Ecto.DateTime.local}
07:05:01.034 [debug] INSERT INTO "domain" ("inserted_at", "updated_at", "url") VALUES ($1, $2, $3) RETURNING "id" [{ {2015, 12, 15}, {7, 5, 0, 0}}, { {2015, 12, 15}, {7, 5, 0, 0}}, "http://about.joekain.com"] OK query=74.2ms queue=2.9ms
%Aggregator.Domain{__meta__: #Ecto.Schema.Metadata<:loaded>, id: 2,
inserted_at: #Ecto.DateTime<2015-12-15T07:05:00Z>,
updated_at: #Ecto.DateTime<2015-12-15T07:05:00Z>,
url: "http://about.joekain.com"}
</code></pre>
</div>
<p>Success!</p>
<h2 id="next-steps">Next steps</h2>
<p>We have our basic setup for Ecto finished. Next we need to make it a little easier to work with and then start doing real work with the model.</p>
<p>In the next post I hope to handle making things easier to work with. This means making those timestamps automatic, and setting up the right aliases and imports to make working Ecto a little less verbose.</p>
<p><a href="http://learningelixir.joekain.com/setting-up-ecto-in-elixir/">Read more...</a></p>
http://learningelixir.joekain.com/idiomatic-elixir2015-12-07T00:00:00-08:002015-12-07T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>What is Idiomatic Elixir?</p>
<p>I think at its core, idiomatic Elixir is what we define it to be. The community, as a collective, defines idiomatic Elixir by writing Elixir code. Idioms are a form of communication, common patterns that we are all familiar with, that we can all recognize.</p>
<p>But, we can only communicate using these patterns if they have wide spread familiarity. To reach this level of communication requires sharing code and reading code written by others.</p>
<p>To this end, I’ve decided to write a book called <em>Idiomatic Elixir.</em> It will be a survey of existing code written by the community, exploring common styles, patterns and techniques.</p>
<p>I’m just getting started with this project, but if you are interested in following the project’s development then I invite you to subscribe to the Idiomatic Elixir mailing list:</p>
<!-- BEGIN: Benchmark Email Signup Form Code -->
<script type="text/javascript" id="lbscript622435" src="https://www.benchmarkemail.com/code/lbformnew.js?mFcQnoBFKMQQhuveDn1m7uZXUawFafLC8RsKgwlboEASQX%252Fl5TlnOA%253D%253D"></script>
<noscript>Please enable JavaScript <br /></noscript>
<!-- END: Benchmark Email Signup Form Code -->
<p>If you subscribe you should expect 2-3 emails per month with progress updates and Elixir tips based on what I learn. You will also get 2 free sample chapters from the book be the first to hear when the book launches.</p>
<p>With that out of the way, let’s look at some idiomatic Elixir.</p>
<h2 id="collectable-idioms">Collectable Idioms</h2>
<p>A few weeks ago I wrote about post about <a href="/learning-elixir-collectable/">the Collectable protocol</a>. I want to continue and look at more examples of how to use <code class="highlighter-rouge">Enum.into</code>.</p>
<h3 id="mapping-maps">Mapping maps</h3>
<p>I’ve mentioned before that <code class="highlighter-rouge">Enum.into/3</code> can be used a lot like like <code class="highlighter-rouge">Enum.map/2</code> in that it can transform data. Below is a great example from Phoenix. The useful thing about this example is that it lets you transform a <code class="highlighter-rouge">Map</code> into another <code class="highlighter-rouge">Map</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defp</span> <span class="n">filter_values</span><span class="p">(%{}</span> <span class="o">=</span> <span class="n">map</span><span class="p">,</span> <span class="n">filter_params</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">into</span> <span class="n">map</span><span class="p">,</span> <span class="p">%{},</span> <span class="k">fn</span> <span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">}</span> <span class="o">-></span>
<span class="k">if</span> <span class="n">is_binary</span><span class="p">(</span><span class="n">k</span><span class="p">)</span> <span class="o">&&</span> <span class="no">String</span><span class="o">.</span><span class="n">contains?</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">filter_params</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="sd">"</span><span class="s2">[FILTERED]"</span><span class="p">}</span>
<span class="k">else</span>
<span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">filter_values</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">filter_params</span><span class="p">)}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>The function used for the transformation solves a specific problem for Phoenix, but that isn’t the purpose of this example. This idiom simplifies down to:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">into</span> <span class="n">map</span><span class="p">,</span> <span class="p">%{},</span> <span class="k">fn</span> <span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">}</span> <span class="o">-></span> <span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">some_function</span><span class="p">(</span><span class="n">v</span><span class="p">)}</span> <span class="k">end</span></code></pre></figure>
<p>This idiom allows you to create a new map with the same keys and transformed values. Of course you can also use a function that remaps the keys, or both the keys and the values.</p>
<p>You can’t use <code class="highlighter-rouge">Enum.map/2</code> to do this because it always returns a list as this test shows:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">Enum.map over a map returns a list"</span> <span class="k">do</span>
<span class="n">map</span> <span class="o">=</span> <span class="p">%{</span><span class="ss">a:</span> <span class="m">1</span><span class="p">,</span> <span class="ss">b:</span> <span class="m">2</span><span class="p">}</span>
<span class="n">result</span> <span class="o">=</span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span> <span class="n">map</span><span class="p">,</span> <span class="k">fn</span> <span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">}</span> <span class="o">-></span> <span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="o">*</span> <span class="m">2</span><span class="p">}</span> <span class="k">end</span>
<span class="n">assert</span> <span class="n">is_list</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<h3 id="collecting-options">Collecting options</h3>
<p>Here’s an example that Phoenix uses twice:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">engines</span> <span class="o">=</span>
<span class="nv">@engines</span>
<span class="o">|></span> <span class="no">Keyword</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">raw_config</span><span class="p">(</span><span class="ss">:template_engines</span><span class="p">))</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="k">fn</span> <span class="p">{</span><span class="n">_</span><span class="p">,</span> <span class="n">v</span><span class="p">}</span> <span class="o">-></span> <span class="n">v</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(%{})</span></code></pre></figure>
<p>This is a useful Elixir idiom for collecting options. It</p>
<ul>
<li>Merges two keyword lists</li>
<li>Filters the lists removing any items with <code class="highlighter-rouge">nil</code> values</li>
<li>Collects the remaining values into a map.</li>
</ul>
<p>This can be used in a lot of places. One would be in collecting options or parameters to a functions. The merge can be used to merge in the default values.</p>
<p>This example starts with a Keyword list but you could also use a <code class="highlighter-rouge">Map</code> for the initial values.</p>
<p>The filter is simple but very useful. The simple function <code class="highlighter-rouge">fn {_, v} -> v end</code> simply checks the value for truthiness. So it will reject key / value pairs with <code class="highlighter-rouge">nil</code> values. This leaves only the meaningful pairs.</p>
<p>The use of <code class="highlighter-rouge">Enum.into/2</code> here is the most simple case but it’s a good ending to the pipeline as it gives us control over the final type of the conversion. If we ended with <code class="highlighter-rouge">Enum.filter/2</code> we would be left with a Keyword List. This is a good example of using <code class="highlighter-rouge">Enumerable</code> types as sources and <code class="highlighter-rouge">Collectable</code> types as sinks.</p>
<h3 id="zip-into">Zip into</h3>
<p>Elixir’s regex module uses:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">zip</span><span class="p">(</span><span class="n">names</span><span class="p">,</span> <span class="n">results</span><span class="p">)</span> <span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(%{})</span></code></pre></figure>
<p>to create a map of named captures and captured results. This is a nice way to create maps if you have separate lists (or enumerables) of keys and values.</p>
<p>Here’s a more concrete example of this technique:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">zip into"</span> <span class="k">do</span>
<span class="n">keys</span> <span class="o">=</span> <span class="p">[</span><span class="ss">:a</span><span class="p">,</span> <span class="ss">:b</span><span class="p">,</span> <span class="ss">:c</span><span class="p">]</span>
<span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="m">1</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">3</span><span class="p">]</span>
<span class="n">result</span> <span class="o">=</span> <span class="no">Enum</span><span class="o">.</span><span class="n">zip</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">values</span><span class="p">)</span> <span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(%{})</span>
<span class="n">assert</span> <span class="n">result</span> <span class="o">==</span> <span class="p">%{</span><span class="ss">a:</span> <span class="m">1</span><span class="p">,</span> <span class="ss">b:</span> <span class="m">2</span><span class="p">,</span> <span class="ss">c:</span> <span class="m">3</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<h3 id="initialization">Initialization</h3>
<p>Elixir has syntactic sugar for creating types like <code class="highlighter-rouge">List</code> and <code class="highlighter-rouge">Map</code> using literals like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">list</span> <span class="o">=</span> <span class="p">[</span><span class="ss">:a</span><span class="p">,</span> <span class="ss">:b</span><span class="p">,</span> <span class="ss">:c</span><span class="p">]</span>
<span class="n">map</span> <span class="o">=</span> <span class="p">%{</span><span class="ss">a:</span> <span class="m">1</span><span class="p">,</span> <span class="ss">b:</span> <span class="m">2</span><span class="p">,</span> <span class="ss">c:</span> <span class="m">3</span><span class="p">}</span></code></pre></figure>
<p>But, some of the more exotic types don’t have a literal syntax and are harder to initialize. For those that implement <code class="highlighter-rouge">Collectable</code>, <code class="highlighter-rouge">Enum.into</code> can be used to create data structures like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="err">~</span><span class="n">w</span><span class="p">(</span><span class="n">foo</span> <span class="n">bar</span> <span class="n">baz</span><span class="p">)</span> <span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="no">MapSet</span><span class="o">.</span><span class="n">new</span><span class="p">)</span></code></pre></figure>
<p>This idiom shows up in many places and I’ve taken this specific example from the <a href="https://github.com/elixir-lang/gettext">gettext</a> library. Any list can be used for initialization of the set types.</p>
<p>This idiom, of course, also works for <code class="highlighter-rouge">HashDict</code> using a Keyword list as the initializer.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">initialize HashDict"</span> <span class="k">do</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[</span><span class="ss">a:</span> <span class="m">1</span><span class="p">,</span> <span class="ss">b:</span> <span class="m">2</span><span class="p">]</span> <span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span> <span class="no">HashDict</span><span class="o">.</span><span class="n">new</span>
<span class="n">assert</span> <span class="no">HashDict</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="ss">:a</span><span class="p">)</span> <span class="o">==</span> <span class="m">1</span>
<span class="n">assert</span> <span class="no">HashDict</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="ss">:b</span><span class="p">)</span> <span class="o">==</span> <span class="m">2</span>
<span class="k">end</span></code></pre></figure>
<h2 id="next-steps">Next Steps</h2>
<p>In this post we looked at several more examples of <code class="highlighter-rouge">Enum.into</code> that show off the usefulness of the <code class="highlighter-rouge">Collectable</code> protocol.</p>
<p>Also, I announced my plans to write <em>Idiomatic Elixir</em> and I hope to fill it with examples like those above, on a variety of different Elixir topics. I hope this post has given you a taste of what the book will contain.</p>
<p><a href="http://learningelixir.joekain.com/idiomatic-elixir/">Read more...</a></p>
http://learningelixir.joekain.com/learning-elixir-with2015-12-01T00:00:00-08:002015-12-01T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>The upcoming release of Elixir 1.2 will introduce a new special form, <code class="highlighter-rouge">with</code>. I wanted to use this post to explore how this new feature works and how to use it effectively going forward.</p>
<p>The purpose of <code class="highlighter-rouge">with</code> is to help chain together commands that return different structured results. One particular use case is to make handling errors a little clearer. When a command’s return value fails to match the <code class="highlighter-rouge">with</code> clause then it falls out of the entire expression. The examples below should help to make this clearer.</p>
<p>Before we dive into the examples, I should note that I’m running against Elixir’s master branch which has support for the <code class="highlighter-rouge">with</code> special form. But, master is closing in on the 1.2 release so you’ll be able to pick up this feature in a stable release soon.</p>
<h2 id="examples-from-the-docs">Examples from the Docs</h2>
<p>First, I want to make sure that I have a working environment with master including <code class="highlighter-rouge">with</code>. As a sanity check I’ll start by copying the examples from the Elixir docs into ExUnit tests.</p>
<p>Here’s the first test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">Example 1"</span> <span class="k">do</span>
<span class="n">opts</span> <span class="o">=</span> <span class="p">%{</span><span class="ss">width:</span> <span class="m">10</span><span class="p">,</span> <span class="ss">height:</span> <span class="m">15</span><span class="p">}</span>
<span class="n">assert</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="m">150</span><span class="p">}</span> <span class="o">==</span>
<span class="n">with</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">width</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Map</span><span class="o">.</span><span class="n">fetch</span><span class="p">(</span><span class="n">opts</span><span class="p">,</span> <span class="ss">:width</span><span class="p">),</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">height</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Map</span><span class="o">.</span><span class="n">fetch</span><span class="p">(</span><span class="n">opts</span><span class="p">,</span> <span class="ss">:height</span><span class="p">),</span>
<span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">width</span> <span class="o">*</span> <span class="n">height</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>Here <code class="highlighter-rouge">with</code> will attempt to match two clauses (the two lines containing <code class="highlighter-rouge"><-</code>). Given <code class="highlighter-rouge">opts</code>, both matches will succeed, the <code class="highlighter-rouge">do</code> block will be executed and its result will be returned. So, the result of the <code class="highlighter-rouge">with</code> expression is <code class="highlighter-rouge"><span class="p">{</span><span class="err">:ok,</span><span class="w"> </span><span class="err">150</span><span class="p">}</span></code> which means that our assertion is true and the test passes.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">Example 2"</span> <span class="k">do</span>
<span class="n">opts</span> <span class="o">=</span> <span class="p">%{</span><span class="ss">width:</span> <span class="m">10</span><span class="p">}</span>
<span class="n">assert</span> <span class="ss">:error</span> <span class="o">==</span>
<span class="n">with</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">width</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Map</span><span class="o">.</span><span class="n">fetch</span><span class="p">(</span><span class="n">opts</span><span class="p">,</span> <span class="ss">:width</span><span class="p">),</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">height</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Map</span><span class="o">.</span><span class="n">fetch</span><span class="p">(</span><span class="n">opts</span><span class="p">,</span> <span class="ss">:height</span><span class="p">),</span>
<span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">width</span> <span class="o">*</span> <span class="n">height</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>This is a modified version of the previous example. We have the same <code class="highlighter-rouge">with</code> expression from Example 1 but different data. In this case there is no <code class="highlighter-rouge">:height</code> key/value pair in <code class="highlighter-rouge">opts</code>.</p>
<p>Because there is no <code class="highlighter-rouge">:height</code> key <code class="highlighter-rouge">Map.fetch/2</code> will return <code class="highlighter-rouge">:error</code> and <code class="highlighter-rouge">with</code> will fail to match the second clause. When <code class="highlighter-rouge">with</code> encounters a clause that doesn’t match it stops evaluation and returns the right hand unmatched value. That means in this case it returns <code class="highlighter-rouge">:error</code>. Hence, our assertion is true and the test passes.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">Example 3"</span> <span class="k">do</span>
<span class="n">width</span> <span class="o">=</span> <span class="no">nil</span>
<span class="n">opts</span> <span class="o">=</span> <span class="p">%{</span><span class="ss">width:</span> <span class="m">10</span><span class="p">,</span> <span class="ss">height:</span> <span class="m">15</span><span class="p">}</span>
<span class="n">assert</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="m">300</span><span class="p">}</span> <span class="o">==</span>
<span class="n">with</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">width</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Map</span><span class="o">.</span><span class="n">fetch</span><span class="p">(</span><span class="n">opts</span><span class="p">,</span> <span class="ss">:width</span><span class="p">),</span>
<span class="n">double_width</span> <span class="o">=</span> <span class="n">width</span> <span class="o">*</span> <span class="m">2</span><span class="p">,</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">height</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Map</span><span class="o">.</span><span class="n">fetch</span><span class="p">(</span><span class="n">opts</span><span class="p">,</span> <span class="ss">:height</span><span class="p">),</span>
<span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">double_width</span> <span class="o">*</span> <span class="n">height</span><span class="p">}</span>
<span class="n">assert</span> <span class="n">width</span> <span class="o">==</span> <span class="no">nil</span>
<span class="k">end</span></code></pre></figure>
<p>The final example from the docs shows the scoping rules used in <code class="highlighter-rouge">with</code> expressions.</p>
<p>The matching behavior is the same as Example 1 but the clauses compute additional values.</p>
<p>We can see that the first clause computes <code class="highlighter-rouge">double_width</code> and that <code class="highlighter-rouge">double_width</code> is in-scope during the <code class="highlighter-rouge">do</code> block.</p>
<p>We can also see from this example that a variable named <code class="highlighter-rouge">width</code> is bound in the first match clause. But, this is not the same variable <code class="highlighter-rouge">width</code> in the scope outside of the the <code class="highlighter-rouge">with</code> expression. That is, variables bound inside the <code class="highlighter-rouge">with</code> expression don’t leak into the outer scope. This is the same as the way the <code class="highlighter-rouge">for</code> special form behaves.</p>
<h2 id="survey-of-the-rfc">Survey of the RFC</h2>
<p>The <code class="highlighter-rouge">with</code> functionality was proposed in the <a href="https://gist.github.com/josevalim/8130b19eb62706e1ab37">Introducing with RFC</a>. I’ll summarize the RFC here to …</p>
<p><code class="highlighter-rouge">with</code> is different than <code class="highlighter-rouge">for</code> in that it matches on values rather than values from collections. There are simple examples and some larger examples. One example I really liked gave a suggested use case for simplifying nested case statements like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">case</span> <span class="no">File</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">path</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">binary</span><span class="p">}</span> <span class="o">-></span>
<span class="k">case</span> <span class="ss">:beam_lib</span><span class="o">.</span><span class="n">chunks</span><span class="p">(</span><span class="n">binary</span><span class="p">,</span> <span class="ss">:abstract_code</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">data</span><span class="p">}</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">wrap</span><span class="p">(</span><span class="n">data</span><span class="p">)}</span>
<span class="n">error</span> <span class="o">-></span>
<span class="n">error</span>
<span class="k">end</span>
<span class="n">error</span> <span class="o">-></span>
<span class="n">error</span>
<span class="k">end</span></code></pre></figure>
<p>into something like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">with</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">binary</span><span class="p">}</span> <span class="o"><-</span> <span class="no">File</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">path</span><span class="p">),</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">data</span><span class="p">}</span> <span class="o"><-</span> <span class="ss">:beam_lib</span><span class="o">.</span><span class="n">chunks</span><span class="p">(</span><span class="n">binary</span><span class="p">,</span> <span class="ss">:abstract_code</span><span class="p">),</span>
<span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">wrap</span><span class="p">(</span><span class="n">data</span><span class="p">)}</span></code></pre></figure>
<p>I find the <code class="highlighter-rouge">with</code> version much more succinct and easy to follow. The nested case version is forced to include a lot of extra and repeated lines for error handling that obscures the intent.</p>
<h2 id="convert-existing-error-handling-code-to-with">Convert existing error handling code to <code class="highlighter-rouge">with</code></h2>
<p>I started paying attention to the <code class="highlighter-rouge">with</code> syntax after searching for good ways to handle errors in Elixir. I found error checking would break up nice pipelines. One way I found was to build monads for describing the error values and then mapping over them.</p>
<p>In a Phoenix based project I’m working on I choose to use a simpler solution. I wrote separate function heads for each of the functions in the chain so that they can pass errors through to the end of the pipeline. This is simple, but can be a bit tedious and the extra function heads obscure the intent a little.</p>
<p>Here’s the code I have:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defp</span> <span class="n">results</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">search_params</span><span class="p">)</span> <span class="k">do</span>
<span class="n">conn</span><span class="o">.</span><span class="n">assigns</span><span class="o">.</span><span class="n">current_user</span>
<span class="o">|></span> <span class="no">Role</span><span class="o">.</span><span class="n">scope</span><span class="p">(</span><span class="ss">can_view:</span> <span class="no">Service</span><span class="p">)</span>
<span class="o">|></span> <span class="n">within</span><span class="p">(</span><span class="n">search_params</span><span class="p">)</span>
<span class="o">|></span> <span class="n">all</span>
<span class="o">|></span> <span class="n">preload</span><span class="p">(</span><span class="ss">:user</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">distance"</span> <span class="o">=></span> <span class="sd">"</span><span class="s2">"</span><span class="p">}),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">distance"</span> <span class="o">=></span> <span class="n">x</span><span class="p">,</span> <span class="sd">"</span><span class="s2">location"</span> <span class="o">=></span> <span class="n">l</span><span class="p">})</span> <span class="k">do</span>
<span class="p">{</span><span class="n">dist</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="no">Float</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="no">Service</span><span class="o">.</span><span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">dist</span><span class="p">,</span> <span class="ss">:miles</span><span class="p">,</span> <span class="n">l</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}</span>
<span class="k">defp</span> <span class="n">all</span><span class="p">({</span><span class="ss">:error</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="n">result</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">result</span>
<span class="k">defp</span> <span class="n">all</span><span class="p">({</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">Repo</span><span class="o">.</span><span class="n">all</span><span class="p">(</span><span class="n">query</span><span class="p">)}</span>
<span class="k">defp</span> <span class="n">preload</span><span class="p">({</span><span class="ss">:error</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="n">result</span><span class="p">,</span> <span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">result</span>
<span class="k">defp</span> <span class="n">preload</span><span class="p">({</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">enum</span><span class="p">},</span> <span class="n">field</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">field</span><span class="p">)}</span>
<span class="k">end</span></code></pre></figure>
<p>I should give a little context.</p>
<p>The <code class="highlighter-rouge">results</code> function is a composition of Ecto queries. It starts with my <code class="highlighter-rouge">User</code> model and queries the services that the user is allowed to view. Then, it limits those services to the set within a given distance of a specified location. This is based on the form parameters. Then, it fetches the services and preloads the users.</p>
<p>The <code class="highlighter-rouge">within</code> function is a wrapper for a function in <code class="highlighter-rouge">Service</code>. It handles two cases that don’t require searching for a distance. If a distance search is required it hands off the work of building the query to <code class="highlighter-rouge">Service.within</code>. This is the function that can fail.</p>
<p>The <code class="highlighter-rouge">all</code> function is part of the error handling case. If <code class="highlighter-rouge">within</code> returned <code class="highlighter-rouge"><span class="p">{</span><span class="err">:error,</span><span class="w"> </span><span class="err">_</span><span class="p">}</span></code> then <code class="highlighter-rouge">all</code> just passes through that error. If <code class="highlighter-rouge">within</code> returns an <code class="highlighter-rouge">:ok</code> value then all queries the <code class="highlighter-rouge">Repo</code>.</p>
<p>The <code class="highlighter-rouge">preload</code> function is similar to <code class="highlighter-rouge">all</code>. It passes through errors, otherwise it calls <code class="highlighter-rouge">Repo.preload</code>.</p>
<p>Using <code class="highlighter-rouge">with</code> I can remove the error cases:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defp</span> <span class="n">results</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">search_params</span><span class="p">)</span> <span class="k">do</span>
<span class="n">with</span> <span class="n">user</span> <span class="o"><-</span> <span class="n">conn</span><span class="o">.</span><span class="n">assigns</span><span class="o">.</span><span class="n">current_user</span><span class="p">,</span>
<span class="n">query</span> <span class="o"><-</span> <span class="no">Role</span><span class="o">.</span><span class="n">scope</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="ss">can_view:</span> <span class="no">Orthrus</span><span class="o">.</span><span class="no">Service</span><span class="p">),</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}</span> <span class="o"><-</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">search_params</span><span class="p">),</span>
<span class="n">query</span> <span class="o"><-</span> <span class="n">all</span><span class="p">(</span><span class="n">query</span><span class="p">),</span>
<span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">preload</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="ss">:user</span><span class="p">)}</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">distance"</span> <span class="o">=></span> <span class="sd">"</span><span class="s2">"</span><span class="p">}),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">distance"</span> <span class="o">=></span> <span class="n">x</span><span class="p">,</span> <span class="sd">"</span><span class="s2">location"</span> <span class="o">=></span> <span class="n">l</span><span class="p">})</span> <span class="k">do</span>
<span class="p">{</span><span class="n">dist</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="no">Float</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="no">Service</span><span class="o">.</span><span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">dist</span><span class="p">,</span> <span class="ss">:miles</span><span class="p">,</span> <span class="n">l</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}</span>
<span class="k">defp</span> <span class="n">all</span><span class="p">(</span><span class="n">query</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">Repo</span><span class="o">.</span><span class="n">all</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
<span class="k">defp</span> <span class="n">preload</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">field</span><span class="p">)</span> <span class="k">do</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">field</span><span class="p">)}</span></code></pre></figure>
<p>Improvements:</p>
<ul>
<li>No longer need extra function heads just for passing errors through to the end</li>
<li>No longer need to pass <code class="highlighter-rouge"><span class="p">{</span><span class="err">:ok,</span><span class="w"> </span><span class="err">term</span><span class="p">}</span></code> form into these functions, <code class="highlighter-rouge">with</code> extracted the values.</li>
<li>No longer need to wrap result in <code class="highlighter-rouge"><span class="p">{</span><span class="nt">"ok, term"</span><span class="err">}</span></code> from <code class="highlighter-rouge">all</code>. This was just
for passthrough</li>
</ul>
<p>I do need to wrap the result of <code class="highlighter-rouge">Repo.preload</code> in <code class="highlighter-rouge"><span class="p">{</span><span class="err">:ok,</span><span class="w"> </span><span class="err">term</span><span class="p">}</span></code> because it
becomes the return value of <code class="highlighter-rouge">results/2</code>. To remove this I would need to do
something with the callers of <code class="highlighter-rouge">results/2</code></p>
<p>Actually, I can take this simplification a step further, there is no reason to wrap <code class="highlighter-rouge">Repo.all</code> anymore at this point. So I can remove the <code class="highlighter-rouge">all</code> function. Also, I can remove the preload function by building the tuple in the do block.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defp</span> <span class="n">results</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">search_params</span><span class="p">)</span> <span class="k">do</span>
<span class="n">with</span> <span class="n">user</span> <span class="o"><-</span> <span class="n">conn</span><span class="o">.</span><span class="n">assigns</span><span class="o">.</span><span class="n">current_user</span><span class="p">,</span>
<span class="n">query</span> <span class="o"><-</span> <span class="no">Role</span><span class="o">.</span><span class="n">scope</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="ss">can_view:</span> <span class="no">Orthrus</span><span class="o">.</span><span class="no">Service</span><span class="p">),</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}</span> <span class="o"><-</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">search_params</span><span class="p">),</span>
<span class="n">query</span> <span class="o"><-</span> <span class="no">Repo</span><span class="o">.</span><span class="n">all</span><span class="p">(</span><span class="n">query</span><span class="p">),</span>
<span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="ss">:user</span><span class="p">)}</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">distance"</span> <span class="o">=></span> <span class="sd">"</span><span class="s2">"</span><span class="p">}),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">distance"</span> <span class="o">=></span> <span class="n">x</span><span class="p">,</span> <span class="sd">"</span><span class="s2">location"</span> <span class="o">=></span> <span class="n">l</span><span class="p">})</span> <span class="k">do</span>
<span class="p">{</span><span class="n">dist</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="no">Float</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="no">Service</span><span class="o">.</span><span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">dist</span><span class="p">,</span> <span class="ss">:miles</span><span class="p">,</span> <span class="n">l</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}</span></code></pre></figure>
<p>I guess there is one more improvement I can make though it doesn’t have anything to do with <code class="highlighter-rouge">with</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defp</span> <span class="n">results</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">search_params</span><span class="p">)</span> <span class="k">do</span>
<span class="n">with</span> <span class="n">user</span> <span class="o"><-</span> <span class="n">conn</span><span class="o">.</span><span class="n">assigns</span><span class="o">.</span><span class="n">current_user</span><span class="p">,</span>
<span class="n">query</span> <span class="o"><-</span> <span class="no">Role</span><span class="o">.</span><span class="n">scope</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="ss">can_view:</span> <span class="no">Orthrus</span><span class="o">.</span><span class="no">Service</span><span class="p">),</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}</span> <span class="o"><-</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">search_params</span><span class="p">),</span>
<span class="n">query</span> <span class="o"><-</span> <span class="no">Repo</span><span class="o">.</span><span class="n">all</span><span class="p">(</span><span class="n">query</span><span class="p">),</span>
<span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="ss">:user</span><span class="p">)}</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="p">%{</span><span class="sd">"</span><span class="s2">distance"</span> <span class="o">=></span> <span class="n">x</span><span class="p">,</span> <span class="sd">"</span><span class="s2">location"</span> <span class="o">=></span> <span class="n">l</span><span class="p">})</span> <span class="k">do</span>
<span class="p">{</span><span class="n">dist</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="no">Float</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="no">Service</span><span class="o">.</span><span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">dist</span><span class="p">,</span> <span class="ss">:miles</span><span class="p">,</span> <span class="n">l</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">within</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">query</span><span class="p">}</span></code></pre></figure>
<p>I’ve removed the first <code class="highlighter-rouge">within/1</code> clause because it was redundant with the
last clause.</p>
<p>At this point, I’m quite happy with the result. The code is so much smaller and in my opinion the intent is much easier to follow.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In this post we read about and played around with the new <code class="highlighter-rouge">with</code> special form. We looked at several examples that explain the feature and its benefits. Then, we used <code class="highlighter-rouge">with</code> to refactor some existing code into a clearer and more concise form.</p>
<p>I have to say, I’m very excited about the new feature. I intend to upgrade my projects to Elixir 1.2 and take advantage of <code class="highlighter-rouge">with</code> to better handle errors in chains of commands.</p>
<p><a href="http://learningelixir.joekain.com/learning-elixir-with/">Read more...</a></p>
http://learningelixir.joekain.com/learning-elixir-collectable2015-11-24T00:00:00-08:002015-11-24T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>In this post I dive into Elixir’s <code class="highlighter-rouge">Collectable</code> protocol. After working through some theory we’ll dig into existing implementations and then implement <code class="highlighter-rouge">Collectable</code> for some of our own types.</p>
<p>Also, if you’ve been following along in my <a href="/elixir-application-design-posts/">series on application architecture</a>, we’ll be adding <code class="highlighter-rouge">Collectable</code> to the <code class="highlighter-rouge">UnshorteningPool</code>. This is something I’ve been talking about doing for some time. If you haven’t been following along in the series, don’t worry you should still be able to follow this article.</p>
<p>The first thing we need to look at is …</p>
<h2 id="enuminto">Enum.into</h2>
<p>The point of <code class="highlighter-rouge">Collectable</code> is to enable <code class="highlighter-rouge">Enum.into/2</code> and <code class="highlighter-rouge">Enum.into/3</code> for different collectable data structures. That is, <code class="highlighter-rouge">Enum.into</code> calls <code class="highlighter-rouge">Collectable.into</code> in order to implement the functionality.</p>
<p>Let’s look at an example of <code class="highlighter-rouge">Enum.into/3</code> - we can use it to transform a list into a <code class="highlighter-rouge">Keyword</code> list, like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="m">1</span><span class="o">..</span><span class="m">10</span><span class="p">,</span> <span class="p">[],</span> <span class="k">fn</span> <span class="n">x</span> <span class="o">-></span> <span class="p">{</span><span class="n">x</span><span class="p">,</span> <span class="n">rem</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="m">2</span><span class="p">)</span> <span class="o">==</span> <span class="m">0</span><span class="p">}</span> <span class="k">end</span><span class="p">)</span></code></pre></figure>
<p>The arguments are</p>
<ol>
<li>An <code class="highlighter-rouge">Enumerable</code>, in our case the range <code class="highlighter-rouge">1..10</code></li>
<li>A <code class="highlighter-rouge">Collectable</code>, in our case a new empty <code class="highlighter-rouge">List</code> <code class="highlighter-rouge">[]</code></li>
<li>A function to transform the data, in our case the anonymous function <code class="highlighter-rouge">fn x -> {x, rem(x, 2) == 0}</code></li>
</ol>
<p>With this setup <code class="highlighter-rouge">Enum.into</code> works just like <code class="highlighter-rouge">Enum.map</code>.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="p">[{</span><span class="m">1</span><span class="p">,</span> <span class="no">false</span><span class="p">},</span> <span class="p">{</span><span class="m">2</span><span class="p">,</span> <span class="no">true</span><span class="p">},</span> <span class="p">{</span><span class="m">3</span><span class="p">,</span> <span class="no">false</span><span class="p">},</span> <span class="p">{</span><span class="m">4</span><span class="p">,</span> <span class="no">true</span><span class="p">},</span> <span class="p">{</span><span class="m">5</span><span class="p">,</span> <span class="no">false</span><span class="p">},</span> <span class="p">{</span><span class="m">6</span><span class="p">,</span> <span class="no">true</span><span class="p">},</span>
<span class="p">{</span><span class="m">7</span><span class="p">,</span> <span class="no">false</span><span class="p">},</span> <span class="p">{</span><span class="m">8</span><span class="p">,</span> <span class="no">true</span><span class="p">},</span> <span class="p">{</span><span class="m">9</span><span class="p">,</span> <span class="no">false</span><span class="p">},</span> <span class="p">{</span><span class="m">10</span><span class="p">,</span> <span class="no">true</span><span class="p">}]</span></code></pre></figure>
<p>Because we’ve written tuples to the array we now have a <code class="highlighter-rouge">Keyword</code> list. In fact, we can enumerate the values <code class="highlighter-rouge">into</code> any <code class="highlighter-rouge">Collectable</code> including a <code class="highlighter-rouge">Map</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">iex</span><span class="p">(</span><span class="m">3</span><span class="p">)</span><span class="o">></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="m">1</span><span class="o">..</span><span class="m">10</span><span class="p">,</span> <span class="p">%{},</span> <span class="k">fn</span> <span class="n">x</span> <span class="o">-></span> <span class="p">{</span><span class="n">x</span><span class="p">,</span> <span class="n">rem</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="m">2</span><span class="p">)</span> <span class="o">==</span> <span class="m">0</span><span class="p">}</span> <span class="k">end</span><span class="p">)</span>
<span class="p">%{</span><span class="m">1</span> <span class="o">=></span> <span class="no">false</span><span class="p">,</span> <span class="m">2</span> <span class="o">=></span> <span class="no">true</span><span class="p">,</span> <span class="m">3</span> <span class="o">=></span> <span class="no">false</span><span class="p">,</span> <span class="m">4</span> <span class="o">=></span> <span class="no">true</span><span class="p">,</span> <span class="m">5</span> <span class="o">=></span> <span class="no">false</span><span class="p">,</span> <span class="m">6</span> <span class="o">=></span> <span class="no">true</span><span class="p">,</span>
<span class="m">7</span> <span class="o">=></span> <span class="no">false</span><span class="p">,</span> <span class="m">8</span> <span class="o">=></span> <span class="no">true</span><span class="p">,</span> <span class="m">9</span> <span class="o">=></span> <span class="no">false</span><span class="p">,</span> <span class="m">10</span> <span class="o">=></span> <span class="no">true</span><span class="p">}</span></code></pre></figure>
<p>The thing to note here is that all <code class="highlighter-rouge">Dict</code> conformant modules accept this tuple form. That is you write 2-tuples into a <code class="highlighter-rouge">Keyword</code>, <code class="highlighter-rouge">HashDict</code> or <code class="highlighter-rouge">Map</code> to build the key-value assocaitions.</p>
<p>You can also use <code class="highlighter-rouge">Enum.into/2</code> which works the same way except that it doesn’t need a function. It passes the values through directly. As an example, this variant can be used to convert a <code class="highlighter-rouge">Keyword</code> list to a <code class="highlighter-rouge">Map</code> or vice-versa:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="c1"># Convert Keyword into Map</span>
<span class="n">iex</span><span class="p">(</span><span class="m">1</span><span class="p">)</span><span class="o">></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">([</span><span class="ss">a:</span> <span class="sd">"</span><span class="s2">b"</span><span class="p">,</span> <span class="ss">b:</span> <span class="sd">"</span><span class="s2">c"</span><span class="p">],</span> <span class="p">%{})</span>
<span class="p">%{</span><span class="ss">a:</span> <span class="sd">"</span><span class="s2">b"</span><span class="p">,</span> <span class="ss">b:</span> <span class="sd">"</span><span class="s2">c"</span><span class="p">}</span>
<span class="c1"># Convert Map into Keyword list</span>
<span class="n">iex</span><span class="p">(</span><span class="m">2</span><span class="p">)</span><span class="o">></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(%{</span><span class="ss">a:</span> <span class="sd">"</span><span class="s2">b"</span><span class="p">,</span> <span class="ss">b:</span> <span class="sd">"</span><span class="s2">c"</span><span class="p">},</span> <span class="p">[])</span>
<span class="p">[</span><span class="ss">a:</span> <span class="sd">"</span><span class="s2">b"</span><span class="p">,</span> <span class="ss">b:</span> <span class="sd">"</span><span class="s2">c"</span><span class="p">]</span></code></pre></figure>
<h2 id="comprehensions">Comprehensions</h2>
<p>Similarly, to <code class="highlighter-rouge">Enum.into/3</code> you can use a for comprehension to store data into an <code class="highlighter-rouge">Collectable</code>. By default a comprehension writes values out to a list but you can use the <code class="highlighter-rouge">into:</code> keyword to use a different type.</p>
<p>We can rewrite our <code class="highlighter-rouge">Map</code> example of <code class="highlighter-rouge">Enum.into/3</code> like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">for</span> <span class="n">x</span> <span class="o"><-</span> <span class="m">1</span><span class="o">..</span><span class="m">10</span><span class="p">,</span> <span class="ss">into:</span> <span class="p">%{}</span> <span class="k">do</span>
<span class="p">{</span><span class="n">x</span><span class="p">,</span> <span class="n">rem</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="m">2</span><span class="p">)</span> <span class="o">==</span> <span class="m">0</span><span class="p">}</span>
<span class="k">end</span></code></pre></figure>
<p>which yields</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="p">%{</span><span class="m">1</span> <span class="o">=></span> <span class="no">false</span><span class="p">,</span> <span class="m">2</span> <span class="o">=></span> <span class="no">true</span><span class="p">,</span> <span class="m">3</span> <span class="o">=></span> <span class="no">false</span><span class="p">,</span> <span class="m">4</span> <span class="o">=></span> <span class="no">true</span><span class="p">,</span> <span class="m">5</span> <span class="o">=></span> <span class="no">false</span><span class="p">,</span> <span class="m">6</span> <span class="o">=></span> <span class="no">true</span><span class="p">,</span>
<span class="m">7</span> <span class="o">=></span> <span class="no">false</span><span class="p">,</span> <span class="m">8</span> <span class="o">=></span> <span class="no">true</span><span class="p">,</span> <span class="m">9</span> <span class="o">=></span> <span class="no">false</span><span class="p">,</span> <span class="m">10</span> <span class="o">=></span> <span class="no">true</span><span class="p">}</span></code></pre></figure>
<p>Again, we generate tuples which fill in the <code class="highlighter-rouge">Map</code>.</p>
<h2 id="protocols">Protocols</h2>
<p>Protocols are the Elixir feature which allow <code class="highlighter-rouge">Enum.into</code> and comprehensions enumerate into different types. A <code class="highlighter-rouge">Protocol</code> defines an abstract interface that can be implemented for different types.</p>
<p>Let’s take a look at the <code class="highlighter-rouge">Collectable</code> <code class="highlighter-rouge">Protocol</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defprotocol</span> <span class="no">Collectable</span> <span class="k">do</span>
<span class="nv">@type</span> <span class="n">command</span> <span class="p">::</span> <span class="p">{</span><span class="ss">:cont</span><span class="p">,</span> <span class="n">term</span><span class="p">}</span> <span class="o">|</span> <span class="ss">:done</span> <span class="o">|</span> <span class="ss">:halt</span>
<span class="nv">@spec</span> <span class="n">into</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">::</span> <span class="p">{</span><span class="n">term</span><span class="p">,</span> <span class="p">(</span><span class="n">term</span><span class="p">,</span> <span class="n">command</span> <span class="o">-></span> <span class="n">t</span> <span class="o">|</span> <span class="n">term</span><span class="p">)}</span>
<span class="k">def</span> <span class="n">into</span><span class="p">(</span><span class="n">collectable</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>I’ve stripped out the documentation, but you can read the full version <a href="https://github.com/elixir-lang/elixir/blob/v1.1.1/lib/elixir/lib/collectable.ex#L1">here</a>.</p>
<p>This looks similar to a <code class="highlighter-rouge">Module</code> with the biggest difference being that we define a function <code class="highlighter-rouge">into/1</code> but it has no body. The fact that there is no body is the point of the <code class="highlighter-rouge">Protocol</code>.</p>
<p>The <code class="highlighter-rouge">Protocol</code> simply defines an interface. It is up an implementation of this protocol to provide a body for the function or functions defined by the <code class="highlighter-rouge">Prototcol</code>.</p>
<p>With that, we should look at implementations of the <code class="highlighter-rouge">Collectable</code> <code class="highlighter-rouge">Protocol</code>.</p>
<h2 id="collectable-implementations"><code class="highlighter-rouge">Collectable</code> implementations</h2>
<p>The <code class="highlighter-rouge">Collectable</code> documentation gives a description of how to implement the <code class="highlighter-rouge">Protocol</code>:</p>
<blockquote>
<p><code class="highlighter-rouge">into(collectable)</code></p>
<p>Returns a function that collects values alongside
the initial accumulation value.</p>
<p>The returned function receives a collectable and injects a given
value into it for every <code class="highlighter-rouge"><span class="p">{</span><span class="err">:cont,</span><span class="w"> </span><span class="err">term</span><span class="p">}</span></code> instruction.</p>
<p><code class="highlighter-rouge">:done</code> is passed when no further values will be injected, useful
for closing resources and normalizing values. A collectable must
be returned on <code class="highlighter-rouge">:done</code>.</p>
<p>If injection is suddenly interrupted, <code class="highlighter-rouge">:halt</code> is passed and it can
return any value, as it won’t be used.</p>
</blockquote>
<p>I have to admit, I found this hard to understand the first few times I read it over. And that’s why I haven’t implemened <code class="highlighter-rouge">Collectable</code> for my <code class="highlighter-rouge">UnshorteningPool</code> (until now). So, let’s look at a few examples to see if that helps us understand what we need to do.</p>
<p>The implementations for <code class="highlighter-rouge">Collectable</code> for the standard library are for <code class="highlighter-rouge">List</code>, <code class="highlighter-rouge">BitString</code> and <code class="highlighter-rouge">Map</code>. They live in the same file referenced above (though there is no requirement that they be in the same file).</p>
<p>Let’s start by taking a look at the implementation for <code class="highlighter-rouge">Map</code></p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defimpl</span> <span class="no">Collectable</span><span class="p">,</span> <span class="ss">for:</span> <span class="no">Map</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">into</span><span class="p">(</span><span class="n">original</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="n">original</span><span class="p">,</span> <span class="k">fn</span>
<span class="n">map</span><span class="p">,</span> <span class="p">{</span><span class="ss">:cont</span><span class="p">,</span> <span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">}}</span> <span class="o">-></span> <span class="ss">:maps</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">,</span> <span class="n">map</span><span class="p">)</span>
<span class="n">map</span><span class="p">,</span> <span class="ss">:done</span> <span class="o">-></span> <span class="n">map</span>
<span class="n">_</span><span class="p">,</span> <span class="ss">:halt</span> <span class="o">-></span> <span class="ss">:ok</span>
<span class="k">end</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>This says that we want to define the implementation of the <code class="highlighter-rouge">Collectable</code> <code class="highlighter-rouge">Protocol</code> for the struct <code class="highlighter-rouge">Map</code>. Within the implementation we get to define <code class="highlighter-rouge">Map</code>’s version of the <code class="highlighter-rouge">into/1</code> function.</p>
<p>The function <code class="highlighter-rouge">into/1</code> returns a 2-tuple. This wasn’t clear to me from the documentation. The two elements of the tuple are</p>
<ol>
<li>A value, this value will be passed back to the function in element 2. You can think of this an accumulator that can be used across multiple invocations of <code class="highlighter-rouge">into/1</code>.</li>
<li>A function that accepts the 3 command terms described in the documentation.</li>
</ol>
<p>So now that we have some sense for the idea of what we are trying to do, let’s look at the specifics for <code class="highlighter-rouge">Map</code>.</p>
<p>The return tuple uses <code class="highlighter-rouge">original</code> as the value. That is, if we had called</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">eleven</span> <span class="o">=</span> <span class="p">{</span><span class="m">11</span> <span class="o">=></span> <span class="no">false</span><span class="p">}</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="m">1</span><span class="o">..</span><span class="m">10</span><span class="p">,</span> <span class="n">eleven</span><span class="p">,</span> <span class="k">fn</span> <span class="n">x</span> <span class="o">-></span> <span class="p">{</span><span class="n">x</span><span class="p">,</span> <span class="n">rem</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="m">2</span><span class="p">)</span> <span class="o">==</span> <span class="m">0</span><span class="p">}</span> <span class="k">end</span><span class="p">)</span></code></pre></figure>
<p>Then <code class="highlighter-rouge">original</code> would be a starting <code class="highlighter-rouge">Map</code>, four. This allows us to start with the passed in map.</p>
<p>The function returned is this 3-headed monster:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">fn</span>
<span class="n">map</span><span class="p">,</span> <span class="p">{</span><span class="ss">:cont</span><span class="p">,</span> <span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">}}</span> <span class="o">-></span> <span class="ss">:maps</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">,</span> <span class="n">map</span><span class="p">)</span>
<span class="n">map</span><span class="p">,</span> <span class="ss">:done</span> <span class="o">-></span> <span class="n">map</span>
<span class="n">_</span><span class="p">,</span> <span class="ss">:halt</span> <span class="o">-></span> <span class="ss">:ok</span>
<span class="k">end</span></code></pre></figure>
<p>There are two arguments in each clause, first the accumulated value. This will be <code class="highlighter-rouge">original</code> the first time the anonymous function is called. The second time, it will be the result of the first call. The value accumulates as we enumerate the source.</p>
<p>The second argument is called the command and it takes one of three forms. We have one function clause per command.</p>
<p>The first clause handles the enumeration case. In this case a single value from the enumeration is passed an in and the anonymous function needs to put it <code class="highlighter-rouge">into</code> the map. It does this simply by calling <code class="highlighter-rouge">:map.put/3</code>.</p>
<p>The second clause handles the completion case. For <code class="highlighter-rouge">Map</code> there is nothing to do. In responsonse to the <code class="highlighter-rouge">:done</code> command the function must return a <code class="highlighter-rouge">Collectable</code> so it returns the accumulated, and now complete, <code class="highlighter-rouge">Map</code>.</p>
<p>The third clause handles error cases and as such can ignore the first parameter (the map). It can also return anything as the value will be ignored. <code class="highlighter-rouge">Map</code> doesn’t need to do any special clean up here but you could imagine that there might be cases that do.</p>
<p>And that’s it for <code class="highlighter-rouge">Map</code>.</p>
<p>Next, let’s look at <code class="highlighter-rouge">List</code>.</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defimpl</span> <span class="no">Collectable</span><span class="p">,</span> <span class="ss">for:</span> <span class="no">List</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">into</span><span class="p">(</span><span class="n">original</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{[],</span> <span class="k">fn</span>
<span class="n">list</span><span class="p">,</span> <span class="p">{</span><span class="ss">:cont</span><span class="p">,</span> <span class="n">x</span><span class="p">}</span> <span class="o">-></span> <span class="p">[</span><span class="n">x</span><span class="o">|</span><span class="n">list</span><span class="p">]</span>
<span class="n">list</span><span class="p">,</span> <span class="ss">:done</span> <span class="o">-></span> <span class="n">original</span> <span class="o">++</span> <span class="ss">:lists</span><span class="o">.</span><span class="n">reverse</span><span class="p">(</span><span class="n">list</span><span class="p">)</span>
<span class="n">_</span><span class="p">,</span> <span class="ss">:halt</span> <span class="o">-></span> <span class="ss">:ok</span>
<span class="k">end</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>This of course takes on the same form as the implementation for <code class="highlighter-rouge">Map</code>. We need another <code class="highlighter-rouge">defimpl</code> block to hold the implementation for <code class="highlighter-rouge">List</code>. Again <code class="highlighter-rouge">into/1</code> returns a tuple which we must look at in more detail.</p>
<p>The return tuple contains <code class="highlighter-rouge">[]</code> as the value. This will be used as an intermediate accumulator.</p>
<p>The <code class="highlighter-rouge"><span class="p">{</span><span class="err">:cont,</span><span class="w"> </span><span class="err">x</span><span class="p">}</span></code> clause accumulates a list. The first time it is called it will construct <code class="highlighter-rouge">[x]</code>. The second time it is called it will prepend the new value to its growing list. By the time the enumeration is done it will have build a list of all items with the last item at the head of this list and the first item at the tail.</p>
<p>At that time the <code class="highlighter-rouge">:done</code> clause will be invoked and the accumulated list will be reversed and appended to the original. This gives the final result for <code class="highlighter-rouge">Enum.into</code> in the expected order.</p>
<p>Again, the <code class="highlighter-rouge">:halt</code> clause doesn’t have much to do in this example.</p>
<div class="notice">
If you like this post then you should know that I'm writing a book full of patterns like these called <i>Idiomatic Elixir</i>.
<br /><br />
If you inerested in the book then sign up below to follow its progress and be notified when it is launched. You'll also receive two free chapters as a sample of what the book will contain.
<br /><br />
Along the way you will also receive Elixir tips based on my research for the book.
<!-- BEGIN: Benchmark Email Signup Form Code -->
<script type="text/javascript" id="lbscript622435" src="https://www.benchmarkemail.com/code/lbformnew.js?mFcQnoBFKMQQhuveDn1m7uZXUawFafLC8RsKgwlboEASQX%252Fl5TlnOA%253D%253D"></script><noscript>Please enable JavaScript <br /></noscript>
<!-- END: Benchmark Email Signup Form Code -->
</div>
<h2 id="implementing-collectable-for-your-own-types">Implementing <code class="highlighter-rouge">Collectable</code> for your own types</h2>
<p>The nice thing about <code class="highlighter-rouge">Protocol</code>s is that you can implement them for your own types. If we implement <code class="highlighter-rouge">Collectable</code> for our own types then we can use <code class="highlighter-rouge">Enum.into</code> to push data into them.</p>
<p>I have two types where I think this might be useful. The first is the <code class="highlighter-rouge">UnshorteningPool</code> from my Domain Scrapper application.</p>
<h3 id="implementing-collectable-for-unshorteningpool">Implementing <code class="highlighter-rouge">Collectable</code> for UnshorteningPool</h3>
<p><code class="highlighter-rouge">UnshorteningPool</code> supports streams of data like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Twitter</span><span class="o">.</span><span class="n">get</span> <span class="o">|></span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">collect</span></code></pre></figure>
<p>By implementing <code class="highlighter-rouge">Collectable</code> I would be able to write:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="no">Twitter</span><span class="o">.</span><span class="n">get</span> <span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="p">)</span></code></pre></figure>
<p>Now, this isn’t any shorter but maybe it’s clearer. I mean, what does <code class="highlighter-rouge">UnshorteningPool.collect/1</code> do? Unless you are familiar with <code class="highlighter-rouge">UnshorteningPool</code>, you may have no idea. The function name <code class="highlighter-rouge">collect</code> is suggestive and is maybe good enough. But, you know what <code class="highlighter-rouge">Enum.into/2</code> does (at least I hope you do now after reading this post). It’s a standard function and communicates the intent without any extra context.</p>
<p>The one roadblock I have is that I actually don’t have a type for <code class="highlighter-rouge">UnshorteningPool</code>. It’s a <code class="highlighter-rouge">GenServer</code> that is identified by its name. But we can make up a type like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UnshorteningPool</span> <span class="k">do</span>
<span class="c1"># Used so we can implement protocols</span>
<span class="n">defstruct</span> <span class="ss">name:</span> <span class="no">UnshorteningPool</span>
<span class="c1"># ...</span>
<span class="k">end</span></code></pre></figure>
<p>This type isn’t really useful for anything. But we can use it as a way to implement protocols. Futhermore, I decided to hide it from the callers so instead they can call <code class="highlighter-rouge">UnshorteningPool.pool</code> to return one these structs:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">pool</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="p">%</span><span class="no">UnshorteningPool</span><span class="p">{}</span></code></pre></figure>
<p>The next step is to implement the protocol. The <code class="highlighter-rouge">UnshorteningPool</code> already has a <code class="highlighter-rouge">Mapper</code> module that handles all the mapping and stream of data in and out of the pool. This is where I want the real code for implementing <code class="highlighter-rouge">into</code> to live. I can do this by writing the <code class="highlighter-rouge">defimpl</code> in it’s own file and then calling into the <code class="highlighter-rouge">Mapper</code> module like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defimpl</span> <span class="no">Collectable</span><span class="p">,</span> <span class="ss">for:</span> <span class="no">UnshorteningPool</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">into</span><span class="p">(%</span><span class="no">UnshorteningPool</span><span class="p">{}),</span> <span class="k">do</span><span class="p">:</span>
<span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Mapper</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">pool_name</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>So, the <code class="highlighter-rouge">into/1</code> function matches on our type and then delegates to <code class="highlighter-rouge">UnshorteningPool.Mapper</code>. Let’s take a look at the new code in the <code class="highlighter-rouge">Mapper</code> module:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">into</span><span class="p">(</span><span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span> <span class="no">nil</span><span class="p">,</span> <span class="o">&</span><span class="n">into</span><span class="p">(</span><span class="n">pool</span><span class="p">,</span> <span class="nv">&1</span><span class="p">,</span> <span class="nv">&2</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">into</span><span class="p">(</span><span class="n">pool</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="p">{</span><span class="ss">:cont</span><span class="p">,</span> <span class="n">item</span><span class="p">}),</span> <span class="k">do</span><span class="p">:</span> <span class="n">push_item_through_pool</span><span class="p">(</span><span class="n">item</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span>
<span class="k">defp</span> <span class="n">into</span><span class="p">(</span><span class="n">_pool</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="ss">:done</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">%</span><span class="no">UnshorteningPool</span><span class="p">{}</span>
<span class="k">defp</span> <span class="n">into</span><span class="p">(</span><span class="n">_pool</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="ss">:halt</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="ss">:ok</span>
<span class="k">defp</span> <span class="n">push_item_through_pool</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="ss">:poolboy</span><span class="o">.</span><span class="n">checkout</span><span class="p">(</span><span class="n">pool</span><span class="p">)</span>
<span class="o">|></span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Worker</span><span class="o">.</span><span class="n">work</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>I added these new functions to implement the <code class="highlighter-rouge">into</code> functionality. First, <code class="highlighter-rouge">into/1</code> builds the return tuple. I don’t need an accumulator since the UnshorteningPool is a server process that maintains its own state so the value in the tuple is <code class="highlighter-rouge">nil</code>. Then, for clarity, I’ve broken out the function in the tuple into a named function in the module, <code class="highlighter-rouge">into/3</code>. I have 3 heads for this function, one for each command.</p>
<p>For <code class="highlighter-rouge"><span class="p">{</span><span class="err">:cont,</span><span class="w"> </span><span class="err">item</span><span class="p">}</span></code> we call <code class="highlighter-rouge">Mapper.push_item_through_pool</code> to do the work of adding new work to the pool. This function checks out a poolboy worker and calls its work function.</p>
<p>For <code class="highlighter-rouge">:done</code> we just return our dummy type. This satisfies the requiremenent that <code class="highlighter-rouge">into/1</code> must return a <code class="highlighter-rouge">Collectable</code>.</p>
<p>For <code class="highlighter-rouge">:halt</code> we don’t do anything special.</p>
<p>Note that all the functions ignore the accumulator argument and that it would be nice to have just not passed it in the first place but I can’t use <code class="highlighter-rouge">&2</code> in a partial without using <code class="highlighter-rouge">&1</code>.</p>
<p>The functionality in <code class="highlighter-rouge">push_item_through_pool/2</code> came from the function <code class="highlighter-rouge">stream_through_pool</code> which used to look like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defp</span> <span class="n">stream_through_pool</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="n">enum</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="n">x</span> <span class="o">-></span> <span class="p">{</span><span class="n">x</span><span class="p">,</span> <span class="ss">:poolboy</span><span class="o">.</span><span class="n">checkout</span><span class="p">(</span><span class="n">pool</span><span class="p">)}</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="p">{</span><span class="n">x</span><span class="p">,</span> <span class="n">worker</span><span class="p">}</span> <span class="o">-></span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Worker</span><span class="o">.</span><span class="n">work</span><span class="p">(</span><span class="n">worker</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">run</span>
<span class="k">end</span></code></pre></figure>
<p>After extracting the functionality from the first two lines I refactored <code class="highlighter-rouge">stream_through_pool</code> into:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defp</span> <span class="n">stream_through_pool</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="o">&</span><span class="n">push_item_through_pool</span><span class="p">(</span><span class="nv">&1</span><span class="p">,</span> <span class="n">pool</span><span class="p">))</span>
<span class="k">end</span></code></pre></figure>
<p>With these changes I have a working into implementation. This test will pass:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">it should implement Collectable"</span> <span class="k">do</span>
<span class="n">input</span> <span class="o">=</span> <span class="p">[</span> <span class="o">...</span> <span class="p">]</span> <span class="c1"># omitted</span>
<span class="n">expected</span> <span class="o">=</span> <span class="p">[</span> <span class="o">...</span><span class="p">]</span> <span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">sort</span> <span class="c1"># omitted</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="n">input</span><span class="p">,</span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">pool</span><span class="p">)</span>
<span class="n">assert</span> <span class="n">expected</span> <span class="o">==</span>
<span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">output_stream</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="m">4</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">sort</span>
<span class="k">end</span></code></pre></figure>
<h3 id="implementing-collectable-for-blockingqueue">Implementing <code class="highlighter-rouge">Collectable</code> for <code class="highlighter-rouge">BlockingQueue</code></h3>
<p>Again, we need to add a new type to use with <code class="highlighter-rouge">Collectable</code> as <code class="highlighter-rouge">BlockingQueue</code> is simply identified by a pid or name. Though BlockingQueue is different from <code class="highlighter-rouge">UnshorteningPool</code> in that there can be more than one. So we do need some data in our new type:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">BlockingQueue</span> <span class="k">do</span>
<span class="n">defstruct</span> <span class="ss">pid:</span> <span class="no">nil</span>
<span class="c1"># ...</span>
<span class="k">end</span></code></pre></figure>
<p>I’ve added a struct for BlockingQueue that includes a field for the <code class="highlighter-rouge">pid</code>. With this we can implement <code class="highlighter-rouge">Collectable</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defimpl</span> <span class="no">Collectable</span><span class="p">,</span> <span class="ss">for:</span> <span class="no">BlockingQueue</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">into</span><span class="p">(%</span><span class="no">BlockingQueue</span><span class="p">{</span><span class="ss">pid:</span> <span class="n">pid</span><span class="p">}),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="no">nil</span><span class="p">,</span> <span class="o">&</span><span class="n">into</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="nv">&1</span><span class="p">,</span> <span class="nv">&2</span><span class="p">)</span> <span class="p">}</span>
<span class="k">defp</span> <span class="n">into</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="p">{</span><span class="ss">:cont</span><span class="p">,</span> <span class="n">item</span><span class="p">}),</span> <span class="k">do</span><span class="p">:</span> <span class="no">BlockingQueue</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="n">item</span><span class="p">)</span>
<span class="k">defp</span> <span class="n">into</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="ss">:done</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">%</span><span class="no">BlockingQueue</span><span class="p">{</span><span class="ss">pid:</span> <span class="n">pid</span><span class="p">}</span>
<span class="k">defp</span> <span class="n">into</span><span class="p">(</span><span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="ss">:halt</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="ss">:ok</span>
<span class="k">end</span></code></pre></figure>
<p>I’ve taken a very similar approach to the one I used for <code class="highlighter-rouge">UnshorteningPool</code> in that I’ve written out the private function <code class="highlighter-rouge">into/3</code> rather than using a large anonymous function. Again, three clauses:</p>
<p>In the <code class="highlighter-rouge"><span class="p">{</span><span class="err">:cont,</span><span class="w"> </span><span class="err">item</span><span class="p">}</span></code> clause I use the existing <code class="highlighter-rouge">BlockingQueue.push/2</code> function to push <code class="highlighter-rouge">item</code> into the queue.</p>
<p>In the <code class="highlighter-rouge">:done</code> clause I don’t need to do anything special. I just return our wrapper type.</p>
<p>In the <code class="highlighter-rouge">:halt</code> clause I don’t need to do anything at all and just return <code class="highlighter-rouge">:ok.</code></p>
<p>Given that I can use the public <code class="highlighter-rouge">push</code> function from <code class="highlighter-rouge">BlockingQueue</code> I don’t need to write a function inside the <code class="highlighter-rouge">BlockingQueue</code> module. So, the whole implementation exists here in collectable.ex.</p>
<p>And with these changes <code class="highlighter-rouge">BlockingQueue</code> can pass a test like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">BlockingQueue should implement Collectable"</span> <span class="k">do</span>
<span class="n">input</span> <span class="o">=</span> <span class="p">[</span><span class="sd">"</span><span class="s2">Hello"</span><span class="p">,</span> <span class="sd">"</span><span class="s2">World"</span><span class="p">]</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">pid</span><span class="p">}</span> <span class="o">=</span> <span class="no">BlockingQueue</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="m">5</span><span class="p">)</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="n">input</span><span class="p">,</span> <span class="p">%</span><span class="no">BlockingQueue</span><span class="p">{</span><span class="ss">pid:</span> <span class="n">pid</span><span class="p">})</span>
<span class="n">assert</span> <span class="n">input</span> <span class="o">==</span> <span class="no">BlockingQueue</span><span class="o">.</span><span class="n">pop_stream</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span> <span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="m">2</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
<p>In this post we dove deep into <code class="highlighter-rouge">Collectable</code>, first looking at how it is used, then how the Elixir standard library implements <code class="highlighter-rouge">Collectable</code> for two of its types and then finally by implementing <code class="highlighter-rouge">Collectable</code> for two types of our own.</p>
<p>I hope you’ve learned a lot from this post; I know I learned a lot in writing it.</p>
<p><a href="http://learningelixir.joekain.com/learning-elixir-collectable/">Read more...</a></p>
http://learningelixir.joekain.com/building-a-cache-in-elixir-with-ets2015-11-17T00:00:00-08:002015-11-17T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>Over the past several weeks I’ve been writing about the <a href="/elixir-application-design-posts/">design and implementation of an Elixir / OTP application</a>. This week, I’m going to step back from the global design and focus on a local design feature for Domain Scrapper by adding a cache to the URL Unshortener using ETS.</p>
<h2 id="erlang-term-storage">Erlang Term Storage</h2>
<p>ETS, or Erlang Term Storage, is a key value store for the Erlang runtime system. We can make use of it to build a cache in Elixir.</p>
<p>I have not used ETS before; I have read about how to build a cache in Saša Jurić’s ‘<a rel="nofollow" href="http://www.amazon.com/gp/product/161729201X/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=161729201X&linkCode=as2&tag=learnelixi-20&linkId=ET5PFLCV6GSDJYYR">Elixir in Action</a><img src="http://ir-na.amazon-adsystem.com/e/ir?t=learnelixi-20&l=as2&o=1&a=161729201X" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" />’. So, given my knowledge but lack of experience I want to do two things while approaching this problem.</p>
<ol>
<li>I want to TDD my solution.</li>
<li>I want to focus first on the interface and integration of the cache into the system and second on the details of ETS.</li>
</ol>
<p>Since ETS is a key / value store it is similar, in interface, to Elixir’s <code class="highlighter-rouge">Map</code>s. I’m familiar with <code class="highlighter-rouge">Map</code> so I want to start by using it as a stand-in for ETS. I want to get a cache working with good tests and integration into my system using a <code class="highlighter-rouge">Map</code> and then switch my implementation over to ETS.</p>
<h2 id="writing-the-tests-and-designing-the-interfaces">Writing the tests and designing the interfaces</h2>
<p>I think of unit tests as the way to design the interface to my module. Then passing the tests becomes an exercise in writing a working implementation of that interface.</p>
<p>Here are the tests I started with:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Cache</span><span class="o">.</span><span class="no">Test</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="n">alias</span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Cache</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">It should not hit when empty"</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="no">Cache</span><span class="o">.</span><span class="n">start_link</span>
<span class="n">assert</span> <span class="no">false</span> <span class="o">==</span> <span class="no">Cache</span><span class="o">.</span><span class="n">check</span><span class="p">(</span><span class="sd">"</span><span class="s2">http:///www.example.com"</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">It should hit after loading a value"</span> <span class="k">do</span>
<span class="no">Cache</span><span class="o">.</span><span class="n">start_link</span>
<span class="no">Cache</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="sd">"</span><span class="s2">http:///www.example.com"</span><span class="p">,</span> <span class="sd">"</span><span class="s2">http://a-long-url.example.com"</span><span class="p">)</span>
<span class="n">assert</span> <span class="sd">"</span><span class="s2">http://a-long-url.example.com"</span> <span class="o">==</span> <span class="no">Cache</span><span class="o">.</span><span class="n">check</span><span class="p">(</span><span class="sd">"</span><span class="s2">http:///www.example.com"</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">It should be able to change a value"</span> <span class="k">do</span>
<span class="no">Cache</span><span class="o">.</span><span class="n">start_link</span>
<span class="no">Cache</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="sd">"</span><span class="s2">http:///www.example.com"</span><span class="p">,</span> <span class="sd">"</span><span class="s2">http://a-long-url.example.com"</span><span class="p">)</span>
<span class="no">Cache</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="sd">"</span><span class="s2">http:///www.example.com"</span><span class="p">,</span> <span class="sd">"</span><span class="s2">http://a-different-url.example.com"</span><span class="p">)</span>
<span class="n">assert</span> <span class="sd">"</span><span class="s2">http://a-different-url.example.com"</span> <span class="o">==</span> <span class="no">Cache</span><span class="o">.</span><span class="n">check</span><span class="p">(</span><span class="sd">"</span><span class="s2">http:///www.example.com"</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">test</span> <span class="sd">"</span><span class="s2">it should register itself with the name UnshorteningPool.Cache"</span> <span class="k">do</span>
<span class="no">Cache</span><span class="o">.</span><span class="n">start_link</span>
<span class="no">Cache</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="sd">"</span><span class="s2">http:///www.example.com"</span><span class="p">,</span> <span class="sd">"</span><span class="s2">http://a-long-url.example.com"</span><span class="p">)</span>
<span class="n">assert</span> <span class="sd">"</span><span class="s2">http://a-long-url.example.com"</span> <span class="o">==</span> <span class="no">Cache</span><span class="o">.</span><span class="n">check</span><span class="p">(</span><span class="sd">"</span><span class="s2">http:///www.example.com"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>These tests describe an interface for a server started with <code class="highlighter-rouge">start_link/0</code>. Once started the server can <code class="highlighter-rouge">check</code> for an existing cache entry or <code class="highlighter-rouge">add</code> a cache entry. <code class="highlighter-rouge">add</code> can either add a new entry or replace an existing one. The server also registers itself with the name UnshorteningPool.Cache.</p>
<p>I chose to use a server to serialize access to the cache itself.</p>
<p>Based on these tests I wrote up this simple GenServer implementation:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Cache</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExActor</span><span class="o">.</span><span class="no">GenServer</span><span class="p">,</span> <span class="ss">export:</span> <span class="bp">__MODULE__</span>
<span class="n">defstart</span> <span class="n">start_link</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">initial_state</span><span class="p">(%{})</span>
<span class="n">defcall</span> <span class="n">check</span><span class="p">(</span><span class="n">url</span><span class="p">),</span> <span class="ss">state:</span> <span class="n">state</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">reply</span><span class="p">(</span><span class="n">state</span><span class="p">[</span><span class="n">url</span><span class="p">]</span> <span class="o">||</span> <span class="no">false</span><span class="p">)</span>
<span class="n">defcast</span> <span class="n">add</span><span class="p">(</span><span class="n">short</span><span class="p">,</span> <span class="n">long</span><span class="p">),</span> <span class="ss">state:</span> <span class="n">state</span> <span class="k">do</span>
<span class="n">new_state</span> <span class="no">Map</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">short</span><span class="p">,</span> <span class="n">long</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Using Saša Jurić’s <a href="https://github.com/sasa1977/exactor">ExActor</a> helps keep our code nice and focused. And we can see that this server is just a simple wrapper over <code class="highlighter-rouge">Map</code>.</p>
<p><code class="highlighter-rouge">start_link/0</code> set’s the server’s initial state to an empty <code class="highlighter-rouge">Map</code>.</p>
<p><code class="highlighter-rouge">check/1</code> looks up an entry from the <code class="highlighter-rouge">Map</code> or returns <code class="highlighter-rouge">false</code> if no entry exists.</p>
<p><code class="highlighter-rouge">add/2</code> is literally a wrapper around <code class="highlighter-rouge">Map.put/2</code>.</p>
<p>This implementation is simple and passes the tests.</p>
<h2 id="integrating-the-cache-into-the-unshortener">Integrating the cache into the unshortener</h2>
<p>The next step is to integrate our new cache into the unshortening process. The cache check belongs in the worker module. It is the worker’s responsibility to unshorten and return results. So it can do the new work of checking the cache. Let’s take a look at what’s currently in the worker and then figure out how to add in a cache check.</p>
<p>The worker is simply:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Worker</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExActor</span><span class="o">.</span><span class="no">GenServer</span>
<span class="n">defstart</span> <span class="n">start_link</span><span class="p">(</span><span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">initial_state</span><span class="p">(</span><span class="m">0</span><span class="p">)</span>
<span class="n">defcast</span> <span class="n">work</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="k">do</span>
<span class="no">BlockingQueue</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">output_queue</span><span class="p">,</span>
<span class="p">{</span><span class="n">self</span><span class="p">,</span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Unshortener</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="n">url</span><span class="p">)})</span>
<span class="n">new_state</span><span class="p">(</span><span class="m">0</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>It is the <code class="highlighter-rouge">work/1</code> cast function that’s important. We can simply add a cache check to it like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">defcast</span> <span class="n">work</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="k">do</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">url</span>
<span class="o">|></span> <span class="n">check_cache</span><span class="p">(</span><span class="k">fn</span> <span class="n">url</span> <span class="o">-></span>
<span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Unshortener</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="k">end</span><span class="p">)</span>
<span class="no">BlockingQueue</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">output_queue</span><span class="p">,</span> <span class="p">{</span><span class="n">self</span><span class="p">,</span> <span class="n">result</span><span class="p">})</span>
<span class="n">new_state</span><span class="p">(</span><span class="m">0</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">check_cache</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span> <span class="k">do</span>
<span class="n">result</span> <span class="o">=</span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Cache</span><span class="o">.</span><span class="n">check</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="k">if</span> <span class="n">!result</span> <span class="k">do</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Cache</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">result</span>
<span class="k">end</span></code></pre></figure>
<p>Here we add a <code class="highlighter-rouge">check_cache/2</code> function which takes the cache key (short url) to be checked and a function. The function computes the value (long URL) to store in the cache in the case of a miss. Hit or miss, <code class="highlighter-rouge">check_cache/2</code> returns the long URL.</p>
<p>That said, I didn’t really like this if/then imperative style when I wrote it. I ended up rewriting it this way:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defp</span> <span class="n">check_cache</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span> <span class="k">do</span>
<span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Cache</span><span class="o">.</span><span class="n">check</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="o">||</span> <span class="n">update_cache</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">update_cache</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span> <span class="k">do</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Cache</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="n">result</span>
<span class="k">end</span></code></pre></figure>
<p>There was one more step required to integrate the cache into my system. I had to start up the <code class="highlighter-rouge">Cache</code> server. I did this by adding it as a worker in my supervision tree in unshortening_pool.ex:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">children</span> <span class="o">=</span> <span class="p">[</span>
<span class="c1"># Define workers and child supervisors to be supervised</span>
<span class="n">worker</span><span class="p">(</span><span class="no">BlockingQueue</span><span class="p">,</span> <span class="p">[</span><span class="ss">:infinity</span><span class="p">,</span> <span class="p">[</span><span class="ss">name:</span> <span class="n">output_queue</span><span class="p">]]),</span>
<span class="n">worker</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Cache</span><span class="p">,</span> <span class="p">[]),</span>
<span class="ss">:poolboy</span><span class="o">.</span><span class="n">child_spec</span><span class="p">(</span><span class="n">pool_name</span><span class="p">(),</span> <span class="n">poolboy_config</span><span class="p">(),</span> <span class="p">[]),</span>
<span class="p">]</span></code></pre></figure>
<p>This change actually broke my tests. And it broke them in two different ways. The first, was that each test called <code class="highlighter-rouge">Cache.start_link</code> to create a new test. This would now fail because the cache was already registered and started up with the application. To fix this I simply removed the calls to <code class="highlighter-rouge">Cache.start_link</code> from the tests. This was a satisfying change in that it actually makes the tests more focused. That is, they focus on the things they say they test and none of the tests claim to test <code class="highlighter-rouge">start_link</code>.</p>
<p>The second way that the tests broke is that the first test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">It should not hit when empty"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="no">false</span> <span class="o">==</span> <span class="no">Cache</span><span class="o">.</span><span class="n">check</span><span class="p">(</span><span class="sd">"</span><span class="s2">http:///www.example.com"</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>continue to fail after removing the call to <code class="highlighter-rouge">start_link</code>. This was due to a race. Because ExUnit runs all my tests in parallel one of the other tests could add the URL “http:///www.example.com” such that this first test would look it up. To fix this I just used a unique URL in this test:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">test</span> <span class="sd">"</span><span class="s2">It should not hit when empty"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="no">false</span> <span class="o">==</span> <span class="no">Cache</span><span class="o">.</span><span class="n">check</span><span class="p">(</span><span class="sd">"</span><span class="s2">http:///not-found.example.com"</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>Arguably, I could change the name of the test as well but I haven’t done that.</p>
<p>With these changes the unit tests pass and when I run the system end-to-end everything works!</p>
<h2 id="elixir-ets">Elixir ETS</h2>
<p>Only at this stage did I start working with ETS.</p>
<p>This was very straight forward, just replacing Map calls with ETS calls:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Cache</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExActor</span><span class="o">.</span><span class="no">GenServer</span><span class="p">,</span> <span class="ss">export:</span> <span class="bp">__MODULE__</span>
<span class="n">defstart</span> <span class="n">start_link</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">initial_state</span><span class="p">(</span><span class="ss">:ets</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="p">[]))</span>
<span class="n">defcall</span> <span class="n">check</span><span class="p">(</span><span class="n">url</span><span class="p">),</span> <span class="ss">state:</span> <span class="n">table</span> <span class="k">do</span>
<span class="k">case</span> <span class="ss">:ets</span><span class="o">.</span><span class="n">lookup</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">url</span><span class="p">)</span> <span class="k">do</span>
<span class="p">[{</span><span class="o">^</span><span class="n">url</span><span class="p">,</span> <span class="n">long</span><span class="p">}]</span> <span class="o">-></span> <span class="n">reply</span><span class="p">(</span><span class="n">long</span><span class="p">)</span>
<span class="p">[]</span> <span class="o">-></span> <span class="n">reply</span><span class="p">(</span><span class="no">false</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">defcast</span> <span class="n">add</span><span class="p">(</span><span class="n">short</span><span class="p">,</span> <span class="n">long</span><span class="p">),</span> <span class="ss">state:</span> <span class="n">table</span> <span class="k">do</span>
<span class="ss">:ets</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="p">{</span><span class="n">short</span><span class="p">,</span> <span class="n">long</span><span class="p">})</span>
<span class="n">new_state</span><span class="p">(</span><span class="n">table</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>The changes are:</p>
<p><code class="highlighter-rouge">start_link/0</code>, instead of setting state to a new <code class="highlighter-rouge">Map</code>, sets the state to a a new ETS table created with <code class="highlighter-rouge">:ets.new/2</code>. I’ve taken the default options by passing <code class="highlighter-rouge">[]</code>. By default the table acts as a set which means a term exists as key only once. Also by default, the access allowed is <code class="highlighter-rouge">protected</code> which means that only the cache server can write to the table. Other processes can read from the table though I don’t make use of this here.</p>
<p><code class="highlighter-rouge">check/1</code>, instead of looking up an entry from the <code class="highlighter-rouge">Map</code>, looks up an entry from the ETS table. The format of the return value is a little different so we use the <code class="highlighter-rouge">case</code> special form to parse it and conform to the <code class="highlighter-rouge">Cache</code>’s return semantics.</p>
<p><code class="highlighter-rouge">add/2</code>, instead of wrapping <code class="highlighter-rouge">Map.put/2</code>, calls <code class="highlighter-rouge">:ets.insert/2</code> to insert a new value in the table.</p>
<p>And that’s it. That’s the extent of the changes to switch from a <code class="highlighter-rouge">Map</code> to ETS. I did’t need to change any of the tests or the integration into the system itself. The tests still pass and when I run the system end-to-end it still works.</p>
<p>There are a few things I could choose to do differently here with respect to the access model for the ETS table. First, I could have set the access to <code class="highlighter-rouge">private</code> instead to prevent other processes from reading directly from the table. This would force access to go through the server. Alternatively, I could choose to change my interface and allow workers to read directly from the ETS table. This would save them a round trip of messages through the <code class="highlighter-rouge">Cache</code> server. It should also reduce the load on the <code class="highlighter-rouge">Cache</code> server. This might be a good change to explore if the <code class="highlighter-rouge">Cache</code> server performance needs improvement.</p>
<h2 id="next-steps">Next Steps</h2>
<p>I was very happy with the results of this iteration of the project. The ETS specific part was actually quite small. I’m happy about the way thing worked out with my plan to start by using a <code class="highlighter-rouge">Map</code> and plugin in ETS at the end.</p>
<p>In my next post I’ll continue refining the local design.</p>
<script type="text/javascript" src="http://wms-na.amazon-adsystem.com/20070822/US/js/link-enhancer-common.js?tag=learnelixi-20&linkId=FK5KWCQXJ5W7NTCQ">
</script>
<noscript>
</noscript>
<p><a href="http://learningelixir.joekain.com/building-a-cache-in-elixir-with-ets/">Read more...</a></p>
http://learningelixir.joekain.com/collecing-multiple-streams-in-elixir2015-11-10T00:00:00-08:002015-11-10T00:00:00-08:00Joseph Kainhttp://learningelixir.joekain.comjoekain@gmail.com
<p>Two weeks ago, I described an <a href="/designing-with-otp-applications-in-elixir/">architecture for Domain Scraper</a> with this diagram</p>
<div style="text-align:center">
<p><img src="http://learningelixir.joekain.com/images/single-pool-flow.png" alt="""" /></p>
</div>
<!--
Fetcher(Twitter) -> \
Fetcher(Reddit) ==> ==> Unshortening Pool ==> Aggregator
Fetcher(HackerNews) -> /
-->
<p>and used it to rationalize the design of putting the various fetchers and the unshortening pool together into separate applications. Then, last week I described how I wrote a new <a href="/fetching-reddit-posts-from-elixir/">Fetcher module for reddit posts</a>.</p>
<p>Now I should be able to put it all together but, as I tried this I found that the current implementation of unshortening pool doesn’t achieve this.</p>
<p>We will correct this as part of this post.</p>
<h2 id="whats-the-problem">What’s the problem?</h2>
<p>Here the runner for our processes:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Main</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">run</span> <span class="k">do</span>
<span class="no">Twitter</span><span class="o">.</span><span class="no">Server</span><span class="o">.</span><span class="n">get</span>
<span class="o">|></span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">map</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="n">x</span> <span class="o">-></span> <span class="no">IO</span><span class="o">.</span><span class="n">inspect</span> <span class="n">x</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="m">10</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>It starts up the server for the Twitter fetchers, maps the Tweets through the <code class="highlighter-rouge">UnshorteningPool</code>, and prints out some of the results. So, I thought to just add another copy of this code to handle reddit. I would somehow put them into separate processes and be done!</p>
<p>But, then I realized that this doesn’t really work the way I want it to. The problem is that I have this one big pipeline which looks like this</p>
<div style="text-align:center">
<p><img src="http://learningelixir.joekain.com/images/twitter-only-flow.png" alt="""" /></p>
</div>
<!--
Fetcher(Twitter) ==> Unshortening Pool ==> Aggregator
-->
<p>But as I mentioned I want this:</p>
<div style="text-align:center">
<p><img src="http://learningelixir.joekain.com/images/single-pool-flow.png" alt="""" /></p>
</div>
<!--
Fetcher(Twitter) -> \
Fetcher(Reddit) ==> ==> Unshortening Pool ==> Aggregator
Fetcher(HackerNews) -> /
-->
<p>So I think I need to be able to write:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">spawn_link</span> <span class="k">fn</span> <span class="o">-></span>
<span class="no">Twitter</span><span class="o">.</span><span class="no">Server</span><span class="o">.</span><span class="n">get</span>
<span class="o">|></span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">map</span>
<span class="k">end</span>
<span class="n">spawn_link</span> <span class="k">fn</span> <span class="o">-></span>
<span class="no">Reddit</span><span class="o">.</span><span class="no">Server</span><span class="o">.</span><span class="n">get</span>
<span class="o">|></span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">map</span>
<span class="k">end</span>
<span class="n">spawn_link</span> <span class="k">fn</span> <span class="o">-></span>
<span class="no">HackerNews</span><span class="o">.</span><span class="no">Server</span><span class="o">.</span><span class="n">get</span>
<span class="o">|></span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">map</span>
<span class="k">end</span>
<span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">output_stream</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="n">x</span> <span class="o">-></span> <span class="no">IO</span><span class="o">.</span><span class="n">inspect</span> <span class="n">x</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="m">30</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span></code></pre></figure>
<p>This would be a change in the interface to <code class="highlighter-rouge">UnshorteningPool</code> so that the <code class="highlighter-rouge">map</code> operation just accepts data and the output <code class="highlighter-rouge">Stream</code> is retrieved in a separate operation (<code class="highlighter-rouge">output_stream/0</code>). And we can no longer use a single pipeline. Given this usage pattern, <code class="highlighter-rouge">map</code> actually doesn’t seem like the right operation anymoreNow, <code class="highlighter-rouge">into</code> is stating to seem like the right abstraction. That is, the servers push into the pool and and we pull the (combined) results out separately. The <code class="highlighter-rouge">map</code> operation makes me thing it should all be one pipeline of computation.</p>
<p>Another way to think about this is, that the design issue is actually in <code class="highlighter-rouge">UnshorteningPool.Mapper</code> in that it creates an output BlockingQueue for each call to <code class="highlighter-rouge">map_through_pool</code>. For my use case I want multiple input <code class="highlighter-rouge">Stream</code>s but only a single output <code class="highlighter-rouge">Stream</code>.</p>
<h2 id="plan">Plan</h2>
<p>My plan to fix this is to restructure UnshorteningPool to</p>
<ul>
<li>Create a single output queue, I can add it to the supervision tree</li>
<li>Replace <code class="highlighter-rouge">map</code> and <code class="highlighter-rouge">map_through_pool</code> with a new interface</li>
<li>Build up processes and output</li>
</ul>
<p>This would let me write something like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Main</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">run</span> <span class="k">do</span>
<span class="n">spawn_link</span> <span class="k">fn</span> <span class="o">-></span>
<span class="no">Twitter</span><span class="o">.</span><span class="no">Server</span><span class="o">.</span><span class="n">get</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">spawn_link</span> <span class="k">fn</span> <span class="o">-></span>
<span class="no">Reddit</span><span class="o">.</span><span class="no">Server</span><span class="o">.</span><span class="n">get</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">spawn_link</span> <span class="k">fn</span> <span class="o">-></span>
<span class="no">HackerNews</span><span class="o">.</span><span class="no">Server</span><span class="o">.</span><span class="n">get</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="p">)</span>
<span class="k">end</span>
<span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">output_stream</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="n">x</span> <span class="o">-></span> <span class="no">IO</span><span class="o">.</span><span class="n">inspect</span> <span class="n">x</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="m">30</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>But, I think there is one more question to answer: where should the processes be spawned?</p>
<p>Previously, <code class="highlighter-rouge">UnshorteningPool.map</code> kicked off a new process. Above, I create them in <code class="highlighter-rouge">run</code>. Ideally, I would create them as workers in the supervision tree so that they could be restarted if anything goes wrong. This means that my <code class="highlighter-rouge">into</code> implementation doesn’t need to spawn. But it does leave us with one more TODO item:</p>
<ul>
<li>Build producer app that supervises all the producers</li>
</ul>
<h2 id="implementation">Implementation</h2>
<p>So, here’s what I ended up with. It’s smaller than what I expected:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Main</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">run</span> <span class="k">do</span>
<span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">output_stream</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="n">x</span> <span class="o">-></span> <span class="no">IO</span><span class="o">.</span><span class="n">inspect</span> <span class="n">x</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="m">30</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p><code class="highlighter-rouge">Main.run</code> just has to take the output of the <code class="highlighter-rouge">UnshorteningPool</code> and for now just print it out.</p>
<p>So you may be wondering where the fetchers are started. Well, in order to have them all supervised they need to go in the supervision tree:</p>
<p>So, there’s a new <code class="highlighter-rouge">Producer</code> app that sets up like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Producer</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Application</span>
<span class="c1"># See http://elixir-lang.org/docs/stable/elixir/Application.html</span>
<span class="c1"># for more information on OTP Applications</span>
<span class="k">def</span> <span class="n">start</span><span class="p">(</span><span class="n">_type</span><span class="p">,</span> <span class="n">_args</span><span class="p">)</span> <span class="k">do</span>
<span class="kn">import</span> <span class="no">Supervisor</span><span class="o">.</span><span class="no">Spec</span><span class="p">,</span> <span class="ss">warn:</span> <span class="no">false</span>
<span class="n">children</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">worker</span><span class="p">(</span><span class="no">Producer</span><span class="o">.</span><span class="no">Worker</span><span class="p">,</span> <span class="p">[</span><span class="no">Reddit</span><span class="o">.</span><span class="no">Server</span><span class="p">],</span> <span class="ss">id:</span> <span class="ss">:producer_reddit</span><span class="p">),</span>
<span class="n">worker</span><span class="p">(</span><span class="no">Producer</span><span class="o">.</span><span class="no">Worker</span><span class="p">,</span> <span class="p">[</span><span class="no">Twitter</span><span class="o">.</span><span class="no">Server</span><span class="p">],</span> <span class="ss">id:</span> <span class="ss">:producer_twitter</span><span class="p">),</span>
<span class="p">]</span>
<span class="c1"># See http://elixir-lang.org/docs/stable/elixir/Supervisor.html</span>
<span class="c1"># for other strategies and supported options</span>
<span class="n">opts</span> <span class="o">=</span> <span class="p">[</span><span class="ss">strategy:</span> <span class="ss">:one_for_one</span><span class="p">,</span> <span class="ss">name:</span> <span class="no">Producer</span><span class="o">.</span><span class="no">Supervisor</span><span class="p">]</span>
<span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>And <code class="highlighter-rouge">Producer.Worker</code> is:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">Producer</span><span class="o">.</span><span class="no">Worker</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExActor</span><span class="o">.</span><span class="no">GenServer</span>
<span class="c1"># Just start up the evaluation</span>
<span class="n">defstart</span> <span class="n">start_link</span><span class="p">(</span><span class="n">module</span><span class="p">)</span> <span class="k">do</span>
<span class="n">pid</span> <span class="o">=</span> <span class="n">spawn_link</span> <span class="k">fn</span> <span class="o">-></span>
<span class="n">module</span><span class="o">.</span><span class="n">get</span>
<span class="o">|></span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">collect</span>
<span class="k">end</span>
<span class="n">initial_state</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>This is a GenServer that takes the place of the anonymous process in the snippet above:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="n">spawn_link</span> <span class="k">fn</span> <span class="o">-></span>
<span class="no">Twitter</span><span class="o">.</span><span class="no">Server</span><span class="o">.</span><span class="n">get</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">into</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>As a GenServer we can install this process into the supervision tree.</p>
<p>Next, let’s look at the changes to <code class="highlighter-rouge">UnshorteningPool</code> that were necessary to support this.</p>
<h2 id="unshortening-pool">Unshortening Pool</h2>
<p>First, we can see hat the new interface to the pool is called <code class="highlighter-rouge">UnshorteningPool.collect/1</code>. Here’s the implementation:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">collect</span><span class="p">(</span><span class="n">enum</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Mapper</span><span class="o">.</span><span class="n">collect</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">pool_name</span><span class="p">)</span></code></pre></figure>
<p>Recall that <code class="highlighter-rouge">UnshorteningPool</code> only provides the top level interfaces but delegates a lot of the work of mapping input to the <code class="highlighter-rouge">UnshorteningPool.Mapper</code> module. This is done to better organize the code. So <code class="highlighter-rouge">collect/1</code> looks up the pool name and passes things on to <code class="highlighter-rouge">Mapper</code>:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defmodule</span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Mapper</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">collect</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">pool</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">stream_through_pool</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span>
<span class="k">def</span> <span class="n">map_through_pool</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="n">enum</span>
<span class="o">|></span> <span class="n">resource</span><span class="p">(</span><span class="n">pool</span><span class="p">)</span>
<span class="o">|></span> <span class="n">extract_and_checkin</span><span class="p">(</span><span class="n">pool</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">resource</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Stream</span><span class="o">.</span><span class="n">resource</span><span class="p">(</span>
<span class="k">fn</span> <span class="o">-></span>
<span class="n">spawn_link</span> <span class="k">fn</span> <span class="o">-></span> <span class="n">stream_through_pool</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span> <span class="k">end</span>
<span class="k">end</span><span class="p">,</span>
<span class="k">fn</span> <span class="n">_</span> <span class="o">-></span> <span class="p">{[</span><span class="no">BlockingQueue</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">output_queue</span><span class="p">)],</span> <span class="no">nil</span><span class="p">}</span> <span class="k">end</span><span class="p">,</span>
<span class="k">fn</span> <span class="n">_</span> <span class="o">-></span> <span class="no">true</span> <span class="k">end</span>
<span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">stream_through_pool</span><span class="p">(</span><span class="n">enum</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="n">enum</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="n">x</span> <span class="o">-></span> <span class="p">{</span><span class="n">x</span><span class="p">,</span> <span class="ss">:poolboy</span><span class="o">.</span><span class="n">checkout</span><span class="p">(</span><span class="n">pool</span><span class="p">)}</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="p">{</span><span class="n">x</span><span class="p">,</span> <span class="n">worker</span><span class="p">}</span> <span class="o">-></span> <span class="no">UnshorteningPool</span><span class="o">.</span><span class="no">Worker</span><span class="o">.</span><span class="n">work</span><span class="p">(</span><span class="n">worker</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">run</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">extract_and_checkin</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Stream</span><span class="o">.</span><span class="n">map</span> <span class="n">stream</span><span class="p">,</span> <span class="k">fn</span> <span class="p">{</span><span class="n">worker</span><span class="p">,</span> <span class="n">result</span><span class="p">}</span> <span class="o">-></span>
<span class="ss">:poolboy</span><span class="o">.</span><span class="n">checkin</span><span class="p">(</span><span class="n">pool</span><span class="p">,</span> <span class="n">worker</span><span class="p">)</span>
<span class="n">result</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">output_stream</span><span class="p">(</span><span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="no">BlockingQueue</span><span class="o">.</span><span class="n">pop_stream</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">output_queue</span><span class="p">)</span>
<span class="o">|></span> <span class="n">extract_and_checkin</span><span class="p">(</span><span class="n">pool</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Here we can see that <code class="highlighter-rouge">Mapper.collect/2</code> reuses the function <code class="highlighter-rouge">stream_through_pool/2</code> which is also used by <code class="highlighter-rouge">map_through_pool</code>. The difference between <code class="highlighter-rouge">collect</code> and <code class="highlighter-rouge">map</code> is is mostly in not building up and returning an output <code class="highlighter-rouge">Stream</code>.</p>
<p>The other significant change to <code class="highlighter-rouge">Mapper</code> is that <code class="highlighter-rouge">map_through_pool</code> used to create a new pool every time. Now, we <code class="highlighter-rouge">UnshorteningPool.output_queue/0</code> to lookup the right queue to use. As we can see:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">output_queue</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="ss">:output_queue</span></code></pre></figure>
<p>this function just returns a globally registered name for the an output queue.</p>
<p>There is also a new function</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">def</span> <span class="n">output_stream</span><span class="p">(</span><span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="no">BlockingQueue</span><span class="o">.</span><span class="n">pop_stream</span><span class="p">(</span><span class="no">UnshorteningPool</span><span class="o">.</span><span class="n">output_queue</span><span class="p">)</span>
<span class="o">|></span> <span class="n">extract_and_checkin</span><span class="p">(</span><span class="n">pool</span><span class="p">)</span>
<span class="k">end</span></code></pre></figure>
<p>We see this function used in <code class="highlighter-rouge">Main.run</code> to retrieve the <code class="highlighter-rouge">Stream</code> of all results from the <code class="highlighter-rouge">UnshorteningPool</code>. This function builds a up the <code class="highlighter-rouge">Stream</code> of results from the queue and then extracts results from the processes and checks the poolboy workers back in, like this:</p>
<figure class="highlight"><pre><code class="language-elixir" data-lang="elixir"><span class="k">defp</span> <span class="n">extract_and_checkin</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">pool</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Stream</span><span class="o">.</span><span class="n">map</span> <span class="n">stream</span><span class="p">,</span> <span class="k">fn</span> <span class="p">{</span><span class="n">worker</span><span class="p">,</span> <span class="n">result</span><span class="p">}</span> <span class="o">-></span>
<span class="ss">:poolboy</span><span class="o">.</span><span class="n">checkin</span><span class="p">(</span><span class="n">pool</span><span class="p">,</span> <span class="n">worker</span><span class="p">)</span>
<span class="n">result</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>This code is also shared with the <code class="highlighter-rouge">map/1</code> path.</p>
<h2 id="next-steps">Next Steps</h2>
<p>This phase of the project went really well. I fixed some interfaces and was then able to assemble everything. The design is looking really good now.</p>
<p>I did need to make a minor change to BlockingQueue to allow it to take a name definition in start_link options. This allowed <code class="highlighter-rouge">UnshorteningPool</code> to use a global registered name for its queue. You can pick this change up in the most recent version of BlockingQueue.</p>
<p>Next, I need to start looking at aggregation of the results. I’ll start on that next week.</p>
<p><a href="http://learningelixir.joekain.com/collecing-multiple-streams-in-elixir/">Read more...</a></p>