Node.js › Message Queues (RabbitMQ & Kafka)
How does Kafka's architecture (topics, partitions, consumer groups) differ fundamentally from RabbitMQ's queue-based model?
AdvancedAnswer
Kafka organizes messages into topics, each split into multiple partitions for parallelism, with messages retained on disk for a configurable period (or indefinitely) rather than being removed once consumed -- this log-based, retained-message model lets multiple independent consumer groups each read the same topic at their own pace, and lets a consumer replay historical messages by resetting its offset, fundamentally different from RabbitMQ's queue model where a message is typically removed once acknowledged and consumed by (usually) one consumer.
const { Kafka } = require('kafkajs');
const kafka = new Kafka({ clientId: 'my-app', brokers: ['localhost:9092'] });
const producer = kafka.producer();
await producer.send({ topic: 'orders', messages: [{ key: String(orderId), value: JSON.stringify(order) }] });
const consumer = kafka.consumer({ groupId: 'order-processors' });
await consumer.subscribe({ topic: 'orders' });
await consumer.run({ eachMessage: async ({ message }) => processOrder(JSON.parse(message.value)) });Real-world example An analytics platform uses Kafka rather than RabbitMQ specifically because it needs both a real-time consumer processing events as they arrive and a separate batch-processing job that periodically replays the last 24 hours of the same event stream from the beginning -- a capability RabbitMQ's queue-based, consume-once model doesn't naturally support.