Building Fault-Tolerant Background Job Systems with Redis and BullMQ

1. Why Background Queues are Non-Negotiable

Executing heavy computational tasks—such as PDF invoice generation, batch email broadcasts, video transcoding, or data exports—directly inside synchronous HTTP handlers blocks client connections and risks server gateway timeouts (504s).

A reliable queue architecture offloads heavy operations to background worker pools, returning an immediate 202 Accepted response with a job tracking ID so the client UI can poll or receive completion notifications via WebSockets.

2. The Redis and BullMQ Architecture

BullMQ leverages Redis atomic Lua scripts, Streams, and sorted sets to implement distributed task scheduling with zero race conditions between concurrent worker processes.

Workers pick up jobs atomically from Redis, update progress percentages, and handle worker crashes gracefully by returning uncompleted jobs back to the active queue after visibility timeouts expire.

import { Worker, Job } from 'bullmq';
import { generatePdfReport } from '@/services/reports';
import { redisConnection } from '@/lib/redis';

interface InvoiceJobData {
  tenantId: string;
  invoiceId: string;
  recipientEmail: string;
}

export const invoiceWorker = new Worker<InvoiceJobData>(
  'invoice-generation',
  async (job: Job<InvoiceJobData>) => {
    job.updateProgress(10);
    // Execute PDF compilation
    const pdfBuffer = await generatePdfReport(job.data.invoiceId);
    job.updateProgress(70);

    // Deliver invoice email with exponential retry support
    await deliverInvoiceEmail(job.data.recipientEmail, pdfBuffer);
    job.updateProgress(100);

    return { status: 'completed', bytes: pdfBuffer.length };
  },
  {
    connection: redisConnection,
    concurrency: 5, // Limit concurrent heavy headless Chromium instances
    limiter: {
      max: 20, // Max 20 email deliveries per minute to avoid provider rate limits
      duration: 60000,
    },
  }
);

3. Idempotency & Exponential Backoff Retries

Transient network failures (such as third-party email provider outages) will cause jobs to fail. Configure exponential backoff retries with jitter to prevent thunderous herd retries from overwhelming recovering downstream systems.

Crucially, job handlers must be strictly idempotent. If a job crashes midway and is re-executed by another worker, it must check database state first to avoid duplicate credit card charges or duplicate customer notifications.

4. Concurrency Throttling & Rate-Limited Workers

Uncontrolled background workers can saturate external third-party API rate limits (e.g. OpenAI rate limits or Stripe API bounds) or exhaust database connection pools.

BullMQ supports distributed token-bucket rate limiters configured directly in Redis, ensuring worker pools automatically throttle throughput across all distributed nodes.

5. Dead-Letter Queues (DLQ) & Worker Telemetry

Jobs that fail after exhausting all retry attempts move into a Dead-Letter Queue (failed state). Telemetry listeners dispatch alerts to Slack/PagerDuty with serialized error stacks.

Engineers can inspect job payloads, correct upstream issues, and replay failed jobs using dashboards like Bull-Board.

6. Frequently Asked Questions