The Elixir Stream Module

It’s been a while since my last post on strict and lazy evaluation. This post continues, as a promised, by looking at Elixir’s standard Stream nodule. Note, there is no github repo for this week’s post. The work for this post was entirely investigative and is shown as excerpts from my iex sessions.

Elixir’s version of streams is quite different from mine so I needed to create some context to guide my reading of the source. I started off with a specific question…

How are streams represented in Elixir?

To answer this at the most basic level I opened up iex and looked at a stream

iex(1)> f = Stream.repeatedly(fn () -> 5 end)
#Function<18.75994740/2 in Stream.repeatedly/1>

So, a stream is a function - that makes sense in terms of the lazy evaluation. Looking at the source for Stream.repeatedly/1 we see that it returns this function

&do_repeatedly(generator_fun, &1, &2)

which is like a partial evaluation of do_repeatedly. Since this function is a stream I figured I should be able to pass it to any of the functions in the Stream module. To verify this I tried it out in iex

iex(3)> f |> Stream.take(5)
#Stream<[enum: #Function<18.75994740/2 in Stream.repeatedly/1>,
 funs: [#Function<42.75994740/1 in Stream.take/2>]]>
iex(4)> f |> Stream.take(5) |> Enum.to_list
[5, 5, 5, 5, 5]

The second command passes f through Stream.take/1 and then through Enum.to_list. I did this just to verify everthing was working properly. It gives the expected result of a list of 5 5’s.

But more interesting, is the first line which only passes f to Stream.take/1. The result here should be another stream. It was interesting that in this case take returns a Stream struct rather than a function. So we are seeing two different representations for streams in Elixir, the first is a function and the second is a struct.

I was curious to see what would happen if I applied Stream.take/1 mulitple times:

iex(7)> f |> Stream.take(5) |> Stream.take(3)
#Stream<[enum: #Function<18.75994740/2 in Stream.repeatedly/1>,
 funs: [#Function<42.75994740/1 in Stream.take/2>,
  #Function<42.75994740/1 in Stream.take/2>]]>

The funs list accumulated another element. I decided to look at the function representation first and save the struct representation a future post. To refocus the investigation I needed a new context. I asked myself…

If a Stream is a function then how can I call that function?

From above we know that Stream.repeatedly/1 returns this function

&do_repeatedly(generator_fun, &1, &2)

where generator_fun is the function passed into Stream.repeatedly/1 (fn () -> 5 end in our example)

Looking at do_repeatedly, I found a multihead function:

defp do_repeatedly(generator_fun, {:suspend, acc}, fun) do
  {:suspended, acc, &do_repeatedly(generator_fun, &1, fun)}
end

defp do_repeatedly(_generator_fun, {:halt, acc}, _fun) do
  {:halted, acc}
end

defp do_repeatedly(generator_fun, {:cont, acc}, fun) do
  do_repeatedly(generator_fun, fun.(generator_fun.(), acc), fun)
end

do_repeatedly matches on the atom in the tuple in the second argumet. The atom must be one of :suspend, :halt, or :cont. I think of this atom as representing the state of the stream’s iteration. The last argument is a function named fun. In the case of state :halt, the function is ignored. For :cont, fun is used to advanced the state with the expectation that the state evaluation eventually goes to :halt. I’ll ignore :suspend for now. Given this, I can build up arguments to the stream function can call it directly:

iex(7)> f.({:cont, 0}, fn (a,acc) -> {:halt, a} end)
{:halted, 5}

I’ve managed to call the function f to evaluate one element (a 5) of the stream and then halt. I’ve passed the required two arguments.

• The first argument {:cont, 0} is used to match the evaluation case of do_repeadly.
• The second argument fn (a,acc) -> {:halt, a} is a simple function that always returns a halting tuple the cause do_repeatedly to stop evaluation after the first iteration.

Upcoming posts

Next week I plan to continue looking at the Elixir Streams implementation by understanding how Streams.map/2 works. The investigation will conintue where we have left off here.

In the following week I plan to go back and understand how the Stream struct works and how it differs form the function representation we looked at in this post.

In the week after I’ll return to Functional Programming in Scala and will work through exercises in Elixir.