Message Passing

tags : Distributed Systems, Inter Process Communication, Concurrency

In my experience of reading articles and discussions related to Message Passing, it’s apparent that the terms are not used very consistently. So don’t make the mistake of being too pedantic about these terms, esp technologies.

Important references §

• Messaging Patterns Overview - Enterprise Integration Patterns
• Messaging patterns - Cloud Design Patterns | Microsoft Learn
• RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1) | Hacker News
• 1024cores - Producer-Consumer Queues
• Anything can be a message queue if you use it wrongly enough - Xe Iaso
• What Kind of Asynchronous is Right For You?

Messaging Patterns §

Direct to Endpoint §

• P2P communications of microservices building a deathstar like architecture
• Everything is linked and effectively you’ll end up with a distributed monolith.
• This is generally not so good
• These 3 blogposts make a good overview about it
  • Why Messaging Queues Suck P1
  • Why Messaging Queues Suck P2
  • Why Messages Queues Might Not Suck

Issues with DoE §

• So basically there are tradefoffs
• Retry logic: in the event of failed consumers/slow consumers
• What guarantees that service is reachable and working as intended?
• Guaranteed delivery?
• What if a lot of systems want to be listening, you’ll have to bake in that logic into the producer. Non-DOE systems help decouple this.

Pub/sub §

• Multiple subscribers can consume the same message from the same topic.

Queue (See Message Queue) §

• For multiple consumers, we need to set up separate queues. Different implementations have different ways of making this happen.

Message Queue (MQ) §

• See Message Queue and Task Queue (different than Message Queus)
• See Orchestrators and Scheduling
• Orchestrators and Job Queus do different things.
  • Eg. Celery is not an orchestrator, it’s a job queue.

Implementations §

Different implementations have different tradefoff, I am just listing random things. Analyze the situation at hand and decide for yourself. Different implementations can support/not support different message passing patterns.

Brokerless §

ZMQ (ZeroMQ) §

• zmq is a communication library, not an application.
• Also useful for communications between services on single server
• It’s like sockets on steroids.

NanoMQ §

Original Nanomsg is abandoned. There’s nng which is a lightweight, broker-less library. People on the internet say it’s inferior to ZMQ.

Broker §

RabbitMQ §

• A message router but can be tricky to administer.
• Supported protocols: AMQP, HTTP, WS, STOMP, MQTT
• Has 1:1, 1:N and others
• Good
  • if retention and streaming is not a requirement (confusing)
  • For long-running tasks
  • A needs to notify another B about something
• Cautionary Tales
  • You probably want to use a wrapper library
  • Watchout for partion_handling
  • Think about how to handle upgrades and restarts etc.
  • Why We Ditched RabbitMQ And Replaced It With A Postgres Queue | Lobsters

Redis §

• See Redis
• The message queue/ pub sub feature of Redis is a secondary use case. Hence, abilities are simpler than RabbitMQ.
• But it can be used as a message queue

Others §

• ActiveMQ
• SQS/SNS
• Google Pub/Sub

Distributed Broker §

Kafka §

• See Kafka
• It’s an event stream processing software/distributed log.
• It is used to stream data in the order it arrives, effectively, in a fault tolerant way.
• It has disk persistence, but not designed for zero data loss
• Misconceptions
  • It’s a broker, not a data lake. (Not designed to batch loads at all times, will affect processing)
  • Your app doesn’t have to be a collection of microservices in order to use Kafka. One can apply CQRS using Kafka on a monolith application but it’s usually done to eventually move towards microservices architecture.
  • Not a message queue
    • You can build queue on top of it, but you need to implement a lot of logic yourself.
    • Durable message queue (due to persistence)

Pulsar §

• Supports all of the messaging patterns such as streaming, pubsub, and queues (Kafka doesn’t support certain queue patterns).

NATS §

• Started as high performance pub/sub
• Now a Kafka competitor w Jetstream(persistent log streams)

NSQ §

• NSQ - A realtime distributed messaging platform
• No builtin support for replication, Looses in-memory things if crashes
• Pub/Sub

Others §

• PostgreSQL via NOTIFY/LISTEN, This is sort of an anti-pattern.
  • Choose Postgres queue technology | Hacker News
  • It’s OKAY to use DB as queue, it is a problem when things scale which is when you can see if incrreasing complexity is worth it
• If you want the single binary with no JVM, Redpanda will likely serve you well.
• Celery (See Task Queue)

Protocol §

See Custom Protocols

AMQP §

• It’s a feature-rich message queue binary protocol, that brokers can use.

MQTT (Protocol) §

• Lightweight message queue protocol especially for embedded devices
• A pub/sub platform