When we generate survey reports for clients, along with online reports with interactive graphs we also generate PDFs that clients can print out and share. Converting our online reports-each of which sometimes has several hundred graphs spread across multiple pages-to PDFs has been an interesting challenge.
We use Sidekiq Pro to manage all background jobs in our Rails application. While Sidekiq has built-in support for prioritizing jobs of different types, the framework does not inherently support prioritization for jobs of the same type. So we built out that feature ourselves!
Our Sidekiq setup involves workers split into a few different worker types differentiated by memory availability and the concurrency with which jobs are processed. Initially, workers belonging to each worker type had a single designated queue they pulled jobs from. For example, consider two types of jobs, FancyJobA and FancyJobB that are both processed by the same type of worker type_a_worker.