Node.js › Message Queues (RabbitMQ & Kafka)

How does Kafka's architecture (topics, partitions, consumer groups) differ fundamentally from RabbitMQ's queue-based model?

Advanced

Answer

Kafka organizes messages into topics, each split into multiple partitions for parallelism, with messages retained on disk for a configurable period (or indefinitely) rather than being removed once consumed -- this log-based, retained-message model lets multiple independent consumer groups each read the same topic at their own pace, and lets a consumer replay historical messages by resetting its offset, fundamentally different from RabbitMQ's queue model where a message is typically removed once acknowledged and consumed by (usually) one consumer.
const { Kafka } = require('kafkajs');

const kafka = new Kafka({ clientId: 'my-app', brokers: ['localhost:9092'] });
const producer = kafka.producer();
await producer.send({ topic: 'orders', messages: [{ key: String(orderId), value: JSON.stringify(order) }] });

const consumer = kafka.consumer({ groupId: 'order-processors' });
await consumer.subscribe({ topic: 'orders' });
await consumer.run({ eachMessage: async ({ message }) => processOrder(JSON.parse(message.value)) });
Real-world example An analytics platform uses Kafka rather than RabbitMQ specifically because it needs both a real-time consumer processing events as they arrive and a separate batch-processing job that periodically replays the last 24 hours of the same event stream from the beginning -- a capability RabbitMQ's queue-based, consume-once model doesn't naturally support.

Common follow-ups: Why is message ordering only guaranteed within a single Kafka partition, not across an entire topic?;How does the choice of a message's partition key affect both ordering guarantees and load distribution across consumers?