Mocking External Dependencies in Elixir

A reader recently asked me about mocking external dependencies in Elixir:

I’m new to Elixir/Phoenix and I’m reviewing the various testing strategies relative to other frameworks I’ve used. Mocking and stubbing are pretty common in Javascript/Python/.NET/Java. Just curious how to best handle external dependencies (e.g., Ecto, modules, etc). I’ve reviewed the behavior/spec approach with Mox, but it’s a little different from defining interfaces and mocking them in other languages.

I come from a Ruby background. So, although I cannot give an answer from a JavaScript, Python, .NET, or Java point of view, I can give one from a Ruby point of view.

Mocking in Elixir is slightly different from doing so in Ruby because we not only have to consider the module and function we want to mock, but we also have to consider its effect across processes.

I want to discuss two popular mocking libraries and how to mock without a library. Let’s look at the three approaches in this order:

• Using the Mock library
• Using vanilla Elixir
• Using the Mox library

We’ll first look at mocking with Mock because I think it will feel familiar. Then we’ll see how to swap modules via application configuration to mock with vanilla Elixir. And finally, we’ll take a look at Mox, which feels like a combination of the best parts of the other two.

Using Mock

Suppose our application uses an external email service. We want to mock our Mailer module so that our tests don’t hit the external API.

Mock allows us to swap a module’s implementation directly in our tests. For example, we can stub our Mailer.deliver/1 function as follows:

defmodule AccountTest do
  use ExUnit.Case, async: false
  import Mock

  test "creates a user account" do
    # stub function
    with_mock Mailer, [deliver: fn(_email) -> :ok end] do
      user = %User{email: "frodo@theshire.com"}
      params = %{user: user}

      account = Account.create(params)

      refute is_nil(account.inserted_at)
    end
  end
end

We replace the regular Mailer.deliver/1 implementation with an anonymous function that takes one argument (which we ignore) and return :ok. The rest of our test is a regular ExUnit test.

Hopefully, that looks familiar. In Ruby, I would write the same test (with RSpec) like this:

RSpec.describe Account do
  it "creates a user account" do
    # stub method
    allow(Mailer).to receive(:deliver)
    user = User.new(email: "frodo@theshire.com")
    account = Account.new(user: user)

    account = account.create

    expect(account.created_at).to be_present
  end
end

Mock also lets us write expectations about the function we’re mocking. That’s particularly helpful when we want to verify side-effects:

defmodule AccountTest do
  use ExUnit.Case, async: false
  import Mock

  test "sends email to user when creating an account" do
    # stub function
    with_mock Mailer, [deliver: fn(_user) -> :ok end] do
      user = %User{email: "frodo@theshire.com"}
      params = %{user: user}

      Account.create(params)

      # confirm function was called with expected arguments
      assert_called Mailer.deliver("frodo@theshire.com")
    end
  end
end

In Ruby, I would write that test like this:

RSpec.describe Account do
  it "sends email to user when creating an account" do
    # stub method
    allow(Mailer).to receive(:deliver)
    user = User.new(email: "frodo@theshire.com")
    account = Account.new(user: user)

    account.create

    # confirm method was called with expected arguments
    expect(Mailer).to have_received(:deliver).with(user.email)
  end
end

As you can see, using Mock is similar to using RSpec, and thus, it probably feels familiar to someone coming from Ruby. But there are downsides to this approach in Elixir.

Downsides

If you notice, the Elixir test module includes use ExUnit.Case, async: false. We need to run our tests synchronously because Mock swaps the module’s implementation globally (across processes). If we ran our test asynchronously, tests in other processes using Mailer.deliver/1 would get the mocked implementation!

Of course, running tests with async: false is not the end of the world. But the more we do that, the slower our tests will be. Still, Mock’s ability to dynamically set stubs and mocks is convenient, and you might consider it worth the trade-off.

The second downside of using Mock is its limited support for making complex assertions. We can use a special wildcard syntax (:_) if we don’t care about the value we pass to Mailer.deliver/1:

# assert Mailer.deliver/1 was called with any argument
assert_called Mailer.deliver(:_)

But if we want to assert something more complex about that argument, we’d have to do a full equality check:

assert_called will check argument equality using == semantics, not pattern matching. For structs, you must provide every property present on the argument as it was called or it will fail.

That can be a bit of a pain. Suppose our Mailer.deliver/1 takes the user struct instead of the email address. We’d have to write our assertion specifying all the fields, regardless of whether or not they are relevant for the test:

# asserting a struct was passed requires all fields
assert_called Mailer.deliver(
  %User{
    first: "Frodo",
    last: "Baggins",
    email: "frodo@theshire.com",
    admin: true,
    inserted_at: ~D[2022-03-29],
    updated_at: ~D[2022-03-29],
    ... more fields!
  }
)

We can use an escape hatch to have more control over assertions, but we have to drop down to meck (the Erlang library Mock is using) to create a new matcher with meck.is/1.

With vanilla Elixir

We can also mock in Elixir without a library. We can create a mock module in our test and swap it for the real one via application configuration.

Let’s revisit our previous test examples. We can write the first one like this:

defmodule AccountTest do
  use ExUnit.Case, async: false

  setup do
    original_module = Application.get_env(:my_app, :mailer)

    on_exit(fn ->
      Application.put_env(:my_app, :mailer, original_module)
    end)
  end

  defmodule MailerMock do
    def deliver(_user) do
      :ok
    end
  end

  test "creates a user account" do
    Application.put_env(:my_app, :mailer, MailerMock)
    user = %User{email: "frodo@theshire.com"}
    params = %{user: user}

    account = Account.create(params)

    refute is_nil(account.inserted_at)
  end
end

That’s a lot more code, so let’s review it. I’ll start with the test and work upwards ⬆️.

• The test injects a MailerMock module dynamically at the beginning via Application.put_env/3. Later, we’ll define Account.create(params) so that it pulls the implementation module via application configuration.

• Going up to the next section, we define the MailerMock module inside the test. It has a single deliver/1 function that returns :ok.

• Finally, we set up on_exit/2 in the setup/1 block to ensure we put back the original module after our test runs. That prevents other tests from accidentally using the fake module we injected.

For this to work, we also have to update our Account module to get the mailer module via Application.get_env/3:

defmodule Account do
  def create(attrs) do
    # do things
    mailer().deliver(email)
  end

  defp mailer do
    Application.get_env(:my_app, :mailer, Mailer)
  end
end

With that set, our test injects the MailerMock, and our Account uses it when calling mailer().deliver(email).

Making assertions about side effects is slightly more complicated. One way to do that is to have our mock module send a message to our test process:

defmodule AccountTest do
  use ExUnit.Case, async: false

  setup do
    original_module = Application.get_env(:my_app, :mailer)

    on_exit(fn ->
      Application.put_env(:my_app, :mailer, original_module)
    end)
  end

  defmodule MailerMock do
    def deliver(user) do
      send(user.test_pid, {:email_delivered, user})
    end
  end

  test "sends email to user when creating an account" do
    Application.put_env(:my_app, :mailer, MailerMock)
    user = %{test_pid: self(), email: "frodo@theshire.com"}
    params = %{user: user}

    Account.create(params)

    assert_receive {:email_delivered, ^user}
  end
end

In that test, we pass the test’s pid as part of the user data (our application would have to allow that), and we use that pid to send a message to the test process when the side effect happens. Finally, our test asserts that it receives the message with the correct user.

Downsides

As you can see, there’s a lot more setup required, but we don’t benefit from it:

• We’re still unable to run our tests asynchronously because we’re changing global application configuration, and

• Making complex assertions is difficult.

What’s more, there are other downsides:

• There’s no clear seam between our internal code and the external service. So, it seems arbitrary that our Account module is responsible for swapping mailer implementations.

• Any other modules that use Mailer will also have to be mocked when tested.

• We don’t have a guarantee that our MailerMock behaves like the real Mailer module.

Using Mox

Mox takes a different approach based on its Mocks and explicit contracts philosophy. It requires that we use mocks as nouns, never as verbs. But it doesn’t stop there. Instead, Mox also requires that we have an explicit contract by which our mocks need to abide.

That is well summarized in the following four points:

1. No ad-hoc mocks. You can only create mocks based on behaviours

2. No dynamic generation of modules during tests. Mocks are preferably defined in your test_helper.exs or in a setup_all block and not per test

3. Concurrency support. Tests using the same mock can still use async: true

4. Rely on pattern matching and function clauses for asserting on the input instead of complex expectation rules

Let’s consider the first two points:

1. No ad-hoc mocks. You can only create mocks based on behaviours.

2. No dynamic generation of modules during tests. Mocks are preferably defined in your test_helper.exs or in a setup_all block and not per test

As you can see, these two points rule out what we did with Mock and vanilla Elixir. Both Mock (and Ruby) used mocking as a verb, swapping the implementation dynamically. And our vanilla Elixir example dynamically created a module in the test.

Instead, with Mox we have to:

• define the mock (noun) ahead of time,
• base our mock on an Elixir behaviour (the contract), and
• swap the real module for the mock module in tests.

That means we have to do a bit of extra plumbing, but I think it’s worth it.

Let’s rewrite our previous examples with Mox. I’ll start by defining our mock in our test helper:

# test/test_helper.exs

Mox.defmock(MailerMock, for: Mailer)

Now, our MailerMock has to satisfy the Mailer behaviour. Let’s define that behaviour:

defmodule Mailer do
  @callback deliver(String.t()) :: :ok | :error
end

We’ll also make the Mailer responsible for pulling the right implementation module from the application configuration:

defmodule Mailer do
  @callback deliver(String.t()) :: :ok | :error

  def deliver(email), do: impl().deliver(email)

  defp impl, do: Application.get_env(:my_app, :mailer, ExternalMailer)
end

Thus, our Mailer module acts as a proxy, forwarding calls to deliver/1 to the implementation module. Mailer sets the default mailer to be our ExternalMailer, which sends real emails.

In our tests, we need to swap the mailer module with our MailerMock. We can do that in our test helper after we’ve defined the mock:

# test/test_helper.exs

Mox.defmock(MailerMock, for: Mailer)
Application.put_env(:my_app, :mailer, MailerMock)

Using application configuration to swap mock modules is similar to what we did in our vanilla Elixir example. But in this case, our Mailer module is an Elixir behaviour, not an ad-hoc module. That means we can always be sure that our MailerMock will define the correct interface for our Mailer. Otherwise, the compiler will warn us.

Furthermore, our vanilla Elixir example had no rhyme or reason as to why our Account module got the mailer implementation from the application configuration. Now, we have a clear seam between our internal code and the external service. The rest of our application doesn’t need to know about any of that. Other modules can simply call Mailer.deliver/1 and move on.

With the setup out of the way, let’s write our tests:

defmodule AccountTest do
  use ExUnit.Case, async: true
  import Mox

  test "creates a user account" do
    MailerMock
    |> stub(:deliver, fn _ -> :ok end)

    user = %User{email: "frodo@theshire.com"}
    params = %{user: user}

    account = Account.create(params)

    refute is_nil(account.inserted_at)
  end

  test "sends email to user when creating an account" do
    MailerMock
    |> expect(:deliver, fn "frodo@theshire.com" -> :ok end)

    user = %User{email: "frodo@theshire.com"}
    params = %{user: user}

    Account.create(params)

    verify!()
  end
end

In the first test, we stub the implementation of MailerMock.deliver/1 with stub/3. In the second test, we mock and define an expectation for MailerMock.deliver/1 with expect/3. Later, we call verify!/0 to ensure the expectation is met.

At this point, using Mox may feel similar to how we used Mock. But we’re doing something very different. With Mock, we were mocking (verb) the Mailer module directly. With Mox, we’re not mocking or stubbing Mailer. Instead, the stub/3 and expect/3 functions define an implementation of deliver/1 for our mock module (MailerMock).

Downsides & Benefits

As you can see, the big downside is that there’s a lot of setup required. So, is the extra setup worth it? That’s where the next two points in Mox’s summary statement come in:

3 Concurrency support. Tests using the same mock can still use async: true

4 Rely on pattern matching and function clauses for asserting on the input instead of complex expectation rules

As you might have noticed, our Mox examples had use ExUnit.Case, async: true! Our tests can run asynchronously because Mox defines the expectations at the process level.

It’s true that we’re swapping the Mailer implementation with MailerMock for all of our tests. But the expectations themselves are defined per process. Thus, tests running in different processes use different expectations!

The second benefit is that we can use pattern matching for expectations. So, if our Mailer.deliver/1 takes an entire user struct, but we only wanted Frodo Baggins to get emails, we can do that with pattern matching!

MailerMock
|> expect(:deliver, fn
  %User{first: "Frodo", last: "Baggins"} -> :ok
  %User{first: "Otho", last: "Sackville-Baggins"} -> :error
end)

When I first came from Ruby, Mock felt more familiar, and it helped make code more pliable. And Mox seemed to have a lot of overhead. But in the end, I like having an explicit boundary when mocking external dependencies (even in Ruby), and I really like the benefits Mox gives us: having asynchronous tests and pattern matching for assertions.

Whether you prefer to use Mox or Mock or something else, I hope understanding the trade-offs helps.

A note on mocking Ecto

The question listed Ecto as one of the external dependencies to mock:

external dependencies (e.g., Ecto, modules, etc). 

For what it’s worth, I never mock Ecto. The database tends to be a core portion of my application, so I include it as part of tests that need it.

Perhaps more importantly, the database is a dependency under my control, and therefore, test setup and expectations are also under my control. That’s very different from a third-party API that sends emails, SMS, etc., where I have little to no control over creating, modifying, and reading data in tests.

And if you want to mock Ecto out of a concern that tests will be too slow because of database operations, I’ve always found my Elixir tests to be fast enough.