Implementing priority lanes for jobs of the same type in Sidekiq

We use Sidekiq Pro to manage all background jobs in our Rails application. While Sidekiq has built-in support for prioritizing jobs of different types, the framework does not inherently support prioritization for jobs of the same type. So we built out that feature ourselves!

Our Sidekiq setup involves workers split into a few different worker types differentiated by memory availability and the concurrency with which jobs are processed. Initially, workers belonging to each worker type had a single designated queue they pulled jobs from. For example, consider two types of jobs, FancyJobA and FancyJobB that are both processed by the same type of worker type_a_worker.

In our Heroku Procfile we mentioned that type_A_worker should pull jobs from a queue called type_A_worker_queue:

However, soon we had 20+ types of jobs that were all processed by the same type of worker as they had similar memory requirements; some of these jobs were more important than others and hence needed to take priority over other jobs in the same queue. As mentioned before, Sidekiq supports this quite well inherently: we were able to configure for each worker type a prioritized set of queues the process pulls jobs from. In our Heroku Procfile we specified:

Also, for each job type we passed to the sidekiq_options hash the appropriate priority queue jobs of that type should go to:

With this, since a type_A_worker would pull off jobs from type_A_worker_queue before it pulled jobs from type_A_worker_low_priority_queue, FancyJobA jobs would be prioritized over FancyJobB jobs. This setup served us well for a while.

Recently, we realized we needed to support different queues for the same type of job depending on where/how the job was queued up. For example, we generate surveys both through an internal admin interface as well as a client-facing interface. While survey generation jobs from our internal admin interface don’t need to be prioritized, those from the client-facing interface cannot stay enqueued for a long time. Sidekiq does not have in-built support for that use case. Our solution has two components:

1. Every job class keeps track of its own worker type

 

2. The process enqueuing a job pushes it to the appropriate queue based on the job’s worker type and priority

When FancyJobA is enqueued, the process enqueuing the job figures out the specific queue to push the job to based on the priority and the job’s WORKER_TYPE constant. In our case, controller actions from our client facing interface enqueue a survey generation job into a higher priority queue for those worker types than jobs enqueued by controller actions from our internal admin interface. This is accomplished using the Sidekiq::Client.push call that takes an optional queue argument. For example, controllers that need to push a low priority FancyJobA call:

In this framework, jobs enqueued by controllers are easy to enqueue to the appropriate queue. However, we hit an interesting bump when we realized we have many instances when a Sidekiq process enqueues other Sidekiq jobs. For these jobs, it is important that they maintained the same priority as the original job the process was executing. For example, if a FancyJobA instance in its perform method calls Sidekiq::Client.push to enqueue another job, the enqueued job should be executed with the same priority as the original FancyJobA job.

To solve this, we wrote Sidekiq server-side middleware that extracts the current job’s priority from its queue name and stores it in a thread local variable to make it globally available when further jobs are being enqueued.

Now, every Sidekiq process has a priority it can access in Thread.current[:job_priority]  and each job’s worker type using the class constant WORKER_TYPE. Using a combination of these, the process knows which queue to push any subsequent job to.

To summarize, this post describes a solution for achieving prioritization for the same type of jobs in Sidekiq, a feature not supported by the framework natively. Hope this was a useful read for those of you who use Sidekiq!

Let us know what you think, and if you like the idea of building on Sidekiq, we’re hiring!

Related Posts
Toward a Swankier Rails Console
Ruby Gem – ExternalFields
Generating PDFs From Webpages With A Large Number Of Graphs