Building reliable background jobs in Ruby on Rails feels like one of those quiet challenges that separates functional applications from truly robust systems. I’ve spent years refining how background processing works, and I’ve learned that the key to stability often lies in a concept called idempotency. An idempotent job can run multiple times without changing the result beyond the initial execution. This matters because jobs retry, networks fail, and queues sometimes deliver the same message more than once.
Let me walk you through some of the most effective patterns I use to make background jobs safe, repeatable, and resilient.
One of the simplest and most powerful techniques involves using unique identifiers for each logical job. Here’s how I often approach it:
class PaymentProcessingJob
include Sidekiq::Worker
sidekiq_options retry: 5
def perform(job_identifier, user_id, amount)
return if already_processed?(job_identifier)
ActiveRecord::Base.transaction do
process_payment(user_id, amount)
mark_as_processed(job_identifier)
end
end
private
def already_processed?(identifier)
Rails.cache.exist?("job_processed_#{identifier}")
end
def mark_as_processed(identifier)
Rails.cache.write("job_processed_#{identifier}", true, expires_in: 48.hours)
end
def process_payment(user_id, amount)
# actual payment gateway call or internal logic
user = User.find(user_id)
user.charge(amount)
end
end
By checking a shared cache before doing any work, I prevent duplicate payments or unintended side effects. The transaction block ensures that the job either completes fully or not at all—no half-finished payments lingering in the system.
Another pattern I frequently rely on is debouncing. This is especially useful when the same event might trigger multiple times in quick succession, like a user mashing a button or an API endpoint receiving redundant calls.
class NotificationJob
include Sidekiq::Worker
def self.debounce(key, wait_time = 10.minutes)
lock_key = "debounce_lock_#{key}"
return if Rails.cache.read(lock_key)
Rails.cache.write(lock_key, true, expires_in: wait_time)
perform_async(key)
end
def perform(key)
# send the notification
User.notify_all(key)
end
end
This way, even if ten events fire at once, only one job goes into the queue. The rest are ignored until the lock expires. It saves resources and prevents users from being spammed with duplicate emails or alerts.
Then there’s the circuit breaker—a pattern I turn to whenever jobs depend on external services that might fail or become unresponsive.
class ExternalApiJob
include Sidekiq::Worker
sidekiq_options retry: 3
def perform(request_data)
if circuit_open?
log_circuit_break
return
end
response = call_external_api(request_data)
reset_failure_count
rescue Timeout::Error, SocketError => e
record_failure
raise e if failure_count < 4
open_circuit
end
private
def circuit_open?
Rails.cache.read('api_circuit_open') == true
end
def open_circuit
Rails.cache.write('api_circuit_open', true, expires_in: 5.minutes)
end
def record_failure
count = Rails.cache.read('failure_count') || 0
Rails.cache.write('failure_count', count + 1, expires_in: 10.minutes)
end
def reset_failure_count
Rails.cache.delete('failure_count')
Rails.cache.delete('api_circuit_open')
end
def call_external_api(data)
# some HTTP call or external service interaction
HTTParty.post('https://api.example.com/process', body: data.to_json)
end
end
When failures pile up, the circuit “breaks,” and subsequent job attempts skip the risky operation entirely. This avoids overwhelming a struggling service and gives it time to recover. I usually pair this with alerting so I know when the circuit has tripped.
Idempotency isn’t just about preventing duplicates—it’s also about designing jobs that can resume gracefully. I often incorporate idempotent receivers when working with webhooks or third-party callbacks.
class WebhookReceiverJob
include Sidekiq::Worker
def perform(event_id, payload)
return if ProcessedEvent.exists?(event_id: event_id)
ActiveRecord::Base.transaction do
event = ProcessedEvent.create!(event_id: event_id)
handle_payload(payload)
end
end
private
def handle_payload(payload)
case payload['type']
when 'invoice.paid'
update_billing(payload['data'])
when 'subscription.updated'
sync_subscription(payload['data'])
end
end
end
By recording each event in the database as it’s processed, I make sure that retries or redeliveries don’t cause double updates. It’s a simple table with a unique constraint on event_id, but it eliminates so many headaches.
For jobs that involve state transitions—like moving an order from “pending” to “completed”—I use optimistic locking or version checks.
class OrderCompletionJob
include Sidekiq::Worker
def perform(order_id)
order = Order.find(order_id)
return unless order.may_complete?
order.transaction do
order.complete!
notify_user(order.user_id)
end
end
end
The may_complete? method checks the current state. If the job runs multiple times, it won’t try to complete an already-completed order. This is simple, readable, and thread-safe.
Finally, I make extensive use of structured logging and metadata in jobs. When something goes wrong, I want to know why—and which specific run of a job encountered the issue.
class LoggableJob
include Sidekiq::Worker
def perform(params)
Rails.logger.tagged(self.class.name, job_id) do
Rails.logger.info "Starting job with #{params}"
execute(params)
Rails.logger.info "Job completed successfully"
end
rescue => e
Rails.logger.error "Job failed: #{e.message}"
raise
end
end
These patterns aren’t just theoretical. I use them every day in production. They reduce support tickets, prevent revenue loss, and let me sleep better at night. Background jobs are the silent workhorses of modern web apps—making them reliable is some of the highest-leverage work you can do.
Remember, the goal isn’t perfection. It’s progress. Start with idempotency keys and debouncing. Add circuit breakers as you integrate with external services. Step by step, you’ll build a system that handles failure not as an exception, but as a normal part of operation.
