Message Queue

tags : Task Queue, Queues, Scheduling and orchestrator, Message Passing

• One of more popular Messaging Patterns
• Async communication on an infrastructural level
• The Queue itself is just a data structure/container and usually needs a Broker (Queue Manager)
• This is different from distributed Task Queue, Message Queue compliment Task queues.

Message Broker (Queue Manager) §

Sort of middleware that can handle things like routing, validate, store, handle backpressure, deliver stuff etc.

Polling §

• User submits a request via the web application
• Backend handles that message by adding a message to the message queue.
• Consumer gets message, makes HTTP call to another web service to actually submit the request.
• From there, the polling logic takes over
• Subsequent messages on the queue each represent a polling attempt to retrieve the results.
• If job has no results, consumer places message back on the queue
  • We can delay the next polling attempt (configurable).
  • Delay logic can use TTL and dead letter definitions etc.

Error Handling §

Fail Fast §

Bail out

Dead Letter Queue §

• Failed messages will be written to a DLQ
• Now you can inspect items in the DLQ and after making any required changes you can now re-process the message from the DLQ
• Almost always needs manual intervention or code to handle stuff written to DLQ