Rails 7.1 adds ActiveJob#perform_all_later to enqueue multiple jobs at once

When using Rails ActiveJob for executing background tasks, you often have to enqueue many of the same jobs simultaneously.

Let's say an application wants to send notifications to all the users.

# app/jobs/send_delivery_notification_job.rb

class SendDeliveryNotificationJob < ApplicationJob
  queue_as :default

  def perform(user_id)
    # code to send a notification to the user
  end
end

# code to send notifications to all users in the system
User.find_each do |user|
  SendDeliveryNotificationJob.perform_later(user.id)
end

Here the application uses Sidekiq as a queuing system, which internally uses Redis for data storage.

In this example, for every user, a call to Redis is made which increases the round-trip latency.

Before Rails 7.1

To avoid this latency issue, Sidekiq, has support for enqueuing multiple jobs at once using the push_bulk method. It helps to reduce the call to the Redis server and cuts out the round-trip latency.

If you are using Sidekiq version < 6.3.0, you can use the push_bulk method as below -

User.find_in_batches do |users|
  user_ids = users.map(&:id)
  Sidekiq::Client.push_bulk('class' => SendDeliveryNotificationJob, 'args' => user_ids)
end

However, if you are using Sidekiq version 6.3.0 or above, you can use the perform_bulk method.

User.find_in_batches do |users|
  user_ids = users.map(&:id)
  SendDeliveryNotificationJob.perform_bulk(user_ids)
end

The Sidekiq author recommends a limit of 1000 jobs per bulk enqueue. It is also the default limit in the perform_bulk method.

Note:

The methods push_bulk and perform_bulk are a part of Sidekiq and now ActiveJob.

In Rails 7.1

Rails 7.1 adds perform_all_later on ActiveJob to enqueue multiple jobs at once.

The perform_all_later method accepts an array of job instances.

Let's refactor the earlier example of sending notifications to use the perform_all_later method.

delivery_notification_jobs = users.map do |user|
  SendDeliveryNotificationJob.new(user.id)
end

ActiveJob.perform_all_later(delivery_notification_jobs)

If you have a situation where you need to enqueue more than 1000 jobs and introduce a delay before executing some of them, you can choose to add a delay for each task.

delivery_notification_jobs = users.map.with_index do |user, index|
  SendDeliveryNotificationJob.new(user.id).set(wait: index.seconds)
end

ActiveJob.perform_all_later(delivery_notification_jobs)

Sidekiq's push_bulk method requires jobs to belong to the same class, the perform_all_later method allows you to enqueue multiple jobs that belong to different classes, providing more flexibility and convenience in managing job queues.

ActiveJob.perform_all_later(
  [
    SendDeliveryNotificationJob.new(user.id),
    OrderDeliveredJob.new(user.id)
  ]
)

Note:

1. If the queuing system does not support bulk enqueuing, perform_all_later will fall back to enqueue each job sequentially. The perform_all_later method does not run any callbacks like before_enqueue, before_perform, after_perform, etc. This is similar to ActiveRecord's bulk import APIs like insert_all, update_all, and upsert_all.

2. Currently, no limit is set on the size of jobs enqueued in one call. The Sidekiq push_bulk method internally handles this case.

3. Unlike perform_later where it returns an instance of the job class queued, the perform_all_later returns nil.

To know more about this feature, please refer to this PR.