Background Jobs & Queues

// Bad: user waits for the entire report generation before getting a response
app.post('/reports', async (req, res) => {
  const report = await generateHugeReport(req.body); // could take minutes
  res.json(report);
});

// Better: enqueue the work and respond immediately
app.post('/reports', async (req, res) => {
  const jobId = await reportQueue.add('generate', req.body);
  res.status(202).json({ jobId, status: 'processing' });
});

Common follow-ups: How does the client find out when a background job has completed, given the response returns before the work is done?;What's the tradeoff between background jobs and simply scaling up the number of server instances handling requests?

const { Queue, Worker } = require('bullmq');
const connection = { host: 'localhost', port: 6379 };

const emailQueue = new Queue('emails', { connection });
await emailQueue.add('welcome-email', { userId: 123 });

// In a separate worker process
new Worker('emails', async (job) => {
  await sendWelcomeEmail(job.data.userId);
}, { connection });

Common follow-ups: How do BullMQ's built-in retry and backoff options compare to implementing retry logic manually?;What happens to jobs already in the queue if the Redis instance backing BullMQ restarts?

// Non-idempotent: retrying this job double-charges the customer
async function chargeCustomer(job) { await paymentGateway.charge(job.data.amount); }

// Idempotent: uses a unique idempotency key so a retry is a no-op
async function chargeCustomer(job) {
  await paymentGateway.charge(job.data.amount, { idempotencyKey: job.data.orderId });
}

Common follow-ups: How do you make a database-write job idempotent when the underlying operation isn't naturally idempotent?;What queue-level guarantees (at-most-once vs at-least-once vs exactly-once) does your specific queue technology actually provide?

const worker = new Worker('orders', processOrder, {
  connection,
  settings: { maxStalledCount: 3 },
});

worker.on('failed', async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    await deadLetterQueue.add('failed-order', { originalJob: job.data, error: err.message });
    await alertOncall(`Job ${job.id} moved to DLQ: ${err.message}`);
  }
});

Common follow-ups: How do you decide an appropriate number of retries before a job should be considered permanently failed and moved to the DLQ?;What operational tooling or dashboard would you build around monitoring and reprocessing a dead-letter queue?

const cron = require('node-cron');

// Runs every day at 2:00 AM
cron.schedule('0 2 * * *', async () => {
  await generateDailyReport();
});

// BullMQ repeatable job, safe across multiple instances
await reportQueue.add('daily-report', {}, { repeat: { pattern: '0 2 * * *' } });

Common follow-ups: Why does a naive node-cron setup break down specifically once an application scales to multiple instances?;How do you handle a scheduled job that's still running when its next scheduled trigger time arrives?

// BullMQ: lower numbers = higher priority
await queue.add('export', premiumUserData, { priority: 1 });
await queue.add('export', freeUserData, { priority: 10 });

// Workers automatically process priority 1 jobs before priority 10 jobs
// when both are waiting in the queue

Common follow-ups: What's the risk of starvation for low-priority jobs if high-priority jobs keep arriving continuously?;How would you combine priority with a separate fairness mechanism to prevent that starvation?

const worker = new Worker('emails', async (job) => {
  await sendEmail(job.data);
}, {
  connection,
  concurrency: 10, // up to 10 email jobs processed simultaneously by this worker
});

Common follow-ups: How do you determine the right concurrency value for a given downstream dependency's actual capacity?;How does concurrency interact with horizontal scaling -- running more worker processes versus raising concurrency within one?

// Job queue (BullMQ): task-oriented, within one application
await imageQueue.add('resize', { imageId, sizes: [100, 500] });

// Message queue (RabbitMQ): event-oriented, across services
channel.publish('orders_exchange', 'order.created', Buffer.from(JSON.stringify(order)));
// Multiple independent services (inventory, shipping, analytics) can each
// subscribe to 'order.created' events without the order service knowing about them

Common follow-ups: Could BullMQ be used for the same inter-service pub/sub use case RabbitMQ handles, and what would be lost by doing so?;How does Kafka's log-based model differ fundamentally from RabbitMQ's queue-based model for this same problem?

process.on('SIGTERM', async () => {
  console.log('Received SIGTERM, closing worker gracefully...');
  await worker.close(); // BullMQ waits for in-progress jobs to finish before resolving
  await connection.quit();
  process.exit(0);
});

Common follow-ups: What happens if the in-progress job takes longer to finish than the orchestrator's configured grace period before it sends SIGKILL?;How do you handle a job that was already 'locked' by a worker that then crashed without a graceful shutdown at all?

// Worker updates progress as it processes
new Worker('video-encode', async (job) => {
  for (let i = 0; i < totalChunks; i++) {
    await encodeChunk(i);
    await job.updateProgress(Math.round((i / totalChunks) * 100));
  }
});

// Client polls for status
app.get('/jobs/:id/status', async (req, res) => {
  const job = await videoQueue.getJob(req.params.id);
  res.json({ progress: job.progress, state: await job.getState() });
});

Common follow-ups: When is Server-Sent Events or WebSockets a better fit than polling for delivering this progress information?;How granular should progress updates be without adding excessive overhead to the job itself?