Elixir background jobs: choosing the right tool for the job

With its vast ecosystem, Elixir offers multiple solutions for running things in the background. To a newcomer or even an experienced developer, the variety of options might seem daunting. Some tools that used to be popular have fallen into the maintenance limbo, while others have emerged as default choices.

This article maps out the Elixir background jobs landscape for you, exploring solutions from the built-in Task module to Redis-backed libraries, such as Exq, and explaining how database-backed systems dominated the ecosystem. Hopefully, this will help you choose the right tool for your project.

Start simple: when Task is enough

Depending on your requirements, you might not need an external solution for your asynchronous job needs. Elixir's standard library offers the Task module, which is perfectly capable of running the code outside of the synchronous execution flow (such as the request-response cycle in a Phoenix app).

Let's see an example:

def create(conn, params) do
  article = Articles.create(params)

  Task.async(fn ->
    EventNotifier.notify(:article_created, %{id: article.id})
  end)

  json(conn, article)
end

In this code example, we are creating an article using Articles context and returning a JSON object representing it. Meanwhile, we also tell the BEAM virtual machine to execute EventNotifier's notification in the background. The rendering process does not need to wait for the notifier to execute. Note that we are not using Task.await/1 here, as this would block the process, and the response to the web request would not be sent immediately.

Tasks can be fired off like the example above shows, but they also can be supervised. The Task documentation strongly suggests putting all tasks under the supervisor, even if we schedule them in a fire-and-forget mode. Doing that creates a clearer expectation of what should happen when the task or a calling process fails.

The supervised version of the notifier code would look like this:

def create(conn, params) do
  article = Articles.create(params)

  {:ok, supervisor} = Task.Supervisor.start_link()
  Task.Supervisor.start_child(supervisor, fn ->
    EventNotifier.notify(:article_created, %{id: article.id})
  end)

  json(conn, article)
end

While Task works well for simple cases, many applications need additional guarantees that it cannot really provide. This is why a whole class of background processing libraries emerged.

Why developers need more than Task

Background job processing systems offer more than just asynchronous jobs. Developers often need additional guarantees and features that simple tools like Task do not provide. These constraints might include:

Durability - once a task is scheduled, it should execute at some point, even if the application goes through a restart or crashes.
Retryability - a system to try again when something goes wrong. This is often paired with an exponential backoff, which means we try in increasingly longer intervals, giving the job more time to execute successfully.
Priorities - when resources are scarce, or a load is huge, it's desired that more important jobs are run before less critical ones.
Other capabilities, such as scheduling cron-like periodic jobs or an ability to ensure uniqueness by inspecting the job data of already scheduled jobs.

Background jobs solutions presented below address these needs and add much more, but they come with their own tradeoffs and challenges.

Redis-backed solutions

The first kind of library we will look into uses Redis as a storage backend. This stems from the early Elixir community branching off from Ruby. In Ruby, and especially in Ruby on Rails, Redis was a widespread addition to every tech stack of every application. Ruby's battle-tested background job solutions, such as Sidekiq and Resque, were built on top of Redis, and so many Elixir solutions took the same path.

Exq

Based on the number of downloads on hex.pm, Exq must be the most popular of the Redis-backed background jobs systems for Elixir. It was created with compatibility with Sidekiq and Resque (Ruby systems I mentioned before), which means that you should be able to schedule a job using Sidekiq and execute it in Exq (or vice-versa). This was a common approach to ease the transition from one language to another.

Exq API is simple. You need to define a module with a perform function and then enqueue a job, referencing the module name as a string:

defmodule MyApp.EmailSender do
  def perform(email, title) do
    # logic for sending an email
  end
end

# enqueuing
{:ok, job_id} = Exq.enqueue(Exq, "queue_name", "MyApp.EmailSender", ["lancelot@avalon.com", "Sir"])

Redis offers strong capabilities for background jobs. Being an in-memory database, it's very fast. It also contains data structures designed specifically for handling queues, which is really useful. However, as Exq's documentation acknowledges, this approach has tradeoffs:

Redis-backed queueing libraries do add additional infrastructure complexity and also overhead due to serialization/marshaling, so make sure to evaluate whether it is an actual need or not.

Exq's last functional release went out in 2022.

Then came Oban

While Redis-backed solutions were prevalent for a good while, the current go-to solution for adding background jobs to the Elixir applications takes a different approach. Oban uses the relational database (PosgreSQL or SQLite) as a storage, which has some interesting tradeoffs:

Eliminates the need for additional databases, as you can just use your main RDBMS to hold your data and power the background jobs.
You get the transactionality for free (sometimes called the outbox pattern).
The database is often a bottleneck in web applications. By tasking it with handling the background jobs, you might incur higher costs or risk worse performance.

You can still have two separate relational databases, one for regular app data and one for Oban. Doing this will solve some problems but will reintroduce others.

The job definition in Oban looks similar to those in Exq et consortes, but with important differences:

defmodule MyApp.EmailSender do
  use Oban.Worker, queue: :email

  @impl true
  def perform(%Oban.Job{args: args}) do
    # do something
  end
end

The main differences are:

Use of macros and behaviours, which we did not see in Eqx, Toniq, and others.
The main perform function takes an Oban.Job struct with a lot of context as the argument instead of an unspecified number of raw parameters. This allows for much better observability and debugging.

The problem of transactionality

What is this transactionality problem I keep referring to? Let's have a look at a concrete example where the issue manifests:

Repo.transaction(fn ->
  foo_result = do_foo()
  bar_result = do_bar()
  schedule_notification(foo_result, bar_result)
  do_baz()
end)

In a traditional Redis-backed system, such as Exq, two things might go wrong here:

If do_baz is slow, the notifications worker might pick up the job before the transaction is committed. It's not uncommon to see errors when an ID of a record created in an earlier step of the transaction is passed to the background job, and then the job reports that it cannot find the record by this ID. This is easily fixed by retires. At some point, the transaction will be committed, and the job will fetch the record successfully. But it at least creates an unnecessary noise in the error tracker.
The second problem is more serious. Imagine that do_baz errors out. The whole transaction is then rolled back, but what happens with the notification? It was already scheduled in a different database and started to live its own life. There is no easy way to unschedule it; most likely, it will be sent, regardless of the fact that the operation that triggered it failed.

How Oban solves it

Oban works in a transactional database and does not hide it. It is especially visible with the API for scheduling the jobs, which is very similar to Ecto API:

%{param1: true, param2: 42}
|> MyApp.NotificationWorker.new()
|> Oban.insert!()

This is, indeed, an insert to the database. So if our schedule_notification/2 above was using Oban:

The job will be invisible to the worker until the transaction is committed.
If the transaction is rolled back, so is the job definition. It will not be performed.

This is, in my opinion, the reason why Oban seems to have won it the Elixir ecosystem. It solved one of the main pain points of the Redis-backed solution and, at the same time, reduced the complexity of the infrastructure stack. While in Ruby on Rails projects, Redis is often considered a default addition to the stack, in Elixir, it's often not needed. But the database is there anyway.

Honorable mention: Que

While Oban reduced infrastructure complexity by eliminating the need for Redis, Que took the simplification idea even further. It's a background job system powered by Mnesia: Erlang's built-in distributed database that comes with every Erlang installation. Now, you don't even need the RDBMS in your app!

While taking an interesting approach, the project was a short-lived one and was abandoned quite quickly, remaining mostly as a curiosity.

Broadway: a different kind of background processing

So far, we've looked at traditional 'fire-and-forget' background job systems. But there's another category of background processing tool in Elixir that serves a different purpose entirely. Broadway, built on top of GenStage, advertises itself like this:

Broadway is a concurrent, multi-stage tool for building data ingestion and data processing pipelines.

The first difference is that you need some kind of queue to get Broadway running. It does not provide it on its own. This queue serves as the source of the events and an entry point to Broadway.

The actual Broadway consists of a bunch of producers, processors, and batchers connected with each other. They compose into a pipeline that ensures that not too many events are processed, they are put into batches as needed, and they execute in parallel if possible.

The image below presents an example Broadway pipeline.

Example of Elixir background jobs processing with Broadway, including one producer, one processor and two batchers

Broadway isn't a background job processor in the traditional sense — it's a data ingestion powerhouse. If your application handles continuous streams of data, needs batching, or integrates with external queues like Kafka or SQS, Broadway might be the right fit. But if you just need to send a welcome email after user registration, stick with Oban.

Your map to Elixir background jobs

We have looked at multiple types of solutions for processing Elixir background jobs, and we learned about their similarities and differences, as well as some historical context. But how would you go about really choosing the right tool for the job? Here's a practical framework to do so:

When in doubt, choose Oban
If you transition from Ruby on Rails application using Sidekiq or Resque, use Exq
If your database is your bottleneck and you don't want to stress it more with background jobs, choose Exq
When you have a complex data processing pipeline (ETL, log processing), consider Broadway

The Elixir ecosystem's evolution toward database-backed solutions reflects the community's focus on simplicity and reliability. By understanding this landscape, you can make informed decisions that will serve your applications well as they grow.

If you want to stay on top of developments like this in the Elixir community, sign up for Honeybadger's newsletter. We will dive deeper into some tools and ideas mentioned in this post.

Elixir background jobs: choosing the right tool for the job

Start simple: when Task is enough

Why developers need more than Task

Redis-backed solutions

Exq

Other Redis-backed solutions

Then came Oban

The problem of transactionality

How Oban solves it

Honorable mention: Que

Broadway: a different kind of background processing

Your map to Elixir background jobs

More articles

Elixir background jobs: choosing the right tool for the job

Start simple: when Task is enough

Why developers need more than Task

Redis-backed solutions

Exq

Other Redis-backed solutions

Then came Oban

The problem of transactionality

How Oban solves it

Honorable mention: Que

Broadway: a different kind of background processing

Your map to Elixir background jobs

More articles

Get Honeybadger's best Elixir articles in your inbox