Message-oriented middleware (MOM) is software or hardware infrastructure supporting sending and receiving messages between distributed systems. The middleware is also known as message broker or message queue. There are many popular message brokers that provide an asynchronous communication protocol. Asynchronous meaning the sender and receiver are completely detached and don’t need to interact with the queueing system simultaneously.
The software is therefore very useful for:
- Decoupling of services
- Painless scaling
- Backpressure management
- Operational visibility
Apache Kafka and RabbitMQ are two of the most popular message brokers and often the question asked by those considering a MOM is ‘Kafka or RabbitMQ?”. Since there is no easy way to answer this, let’s analyze the strengths of each and also the differences.
RabbitMQ is a popular message broker with many powerful features. Their website features excellent documentation and there are also many books on the subject. It is written in Erlang (not a well-known programming language but perfect for the task). RabbitMQ has been developed and maintained by Pivotal. It is easy to use and supports a number of development platforms. In short, it is a robust, mature, general purpose message broker that supports many standard protocols such as AMQP.
RabbitMQ has features that let you trade off performance with reliability, including persistence, delivery acknowledgements, publisher confirms, and high availability.
2. Flexible Routing
Messages are routed through exchanges before landing at queues. RabbitMQ has many built-in exchange types for typical routing logic. For more complex routing, combine exchanges together or write your own exchange type as a plugin.
Multiple RabbitMQ servers on the same local network can be grouped together to form a single logical broker.
RabbitMQ provides a federation model for servers that need to be more loosely coupled and unreliably connected than clustering allows.
5. Highly Available Queues
It has capability to mirror queues across several machines in a cluster to ensure that messages are secure in the event of a hardware failure.
It supports messaging over a variety of messaging protocols.
Additionally, there are many other features like supporting multiple clients, simpler management UI, tracing support for messages, and a variety of plugins that are extensible. RabbitMQ can work well for “fast” consumers, if overall throughput is enough for your requirements.
Kafka is a distributed streaming platform. It is used for real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, fast, and runs in production in thousands of companies. Think of it as a message bus optimized for high-ingress data streams and replay.
1. Kafka lets you publish and subscribe to streams of records. Here it behaves similar to a message queue or enterprise messaging system.
2. It allows you to store streams of records in a fault-tolerant way.
3. It lets you process streams of records as they occur.
4. Kafka is excellent for building real-time streaming data pipelines that reliably get data between systems or applications. It is also good for building real-time streaming applications that transform or react to the streams of data
5. Kafka is a highly scalable, robust, high performance and a durable messaging system.
Differences between the two:
- RabbitMQ is more solid, mature and a general purpose message broker that supports several standardized protocols such as AMQP
- High-throughput and reliable background jobs, communication and integration within, and between applications.
- Open Source: Mozilla Public License
- Erlang: Programming Language
- Many mature libraries including: Ruby, Python, Node.js, Clojure, Go, Java and C
- Apache Kafka is a message bus optimized for high-ingress data streams and replay
- Distributed streaming platform used for real-time data pipelines. Build applications that process and re-process streamed data on disk
- Open Source: Apache License 2.0
- Scala (JVM): Programming Language
- Many including: Ruby, Python, Node.js and Java
RabbitMQ or Kafka?
The answer to this question is ‘it depends’. It depends on the use-case and specific message broking requirements. Message brokers/MOM are a very broad and complex topic, and we have just scratched the surface. However, if you have to take a call on which message broking system to use, here is a handy tip:
- Kafka would work better for “fast” and reliable consumers
- RabbitMQ might be the answer for “slow” and unreliable consumers
Use your own discretion and research before you take the call as no two requirements are alike. If you would like to customize your message distribution system then you can use the services of a specialized Java enterprise development company like Rishabh Software. We would like to know your views on message brokers and whether you find one favorable over the other. Please let us know in the comments section below.