0

Imagine a queue system with different consumers - some are low cost whist (like 'reserved/spot' instances in the AWS world) others would be high cost.

The system should ensure the messages are processed by the low cost consumers when available, but if they are unable to process a message after a certain time period then they should be processed by the high cost consumers.

I imagine this could be done by something like

  • A queue that moved messages onto another queue if they haven't been processed by X time.
  • A delay queue where the delay can be overridden on 'get'. (So the low cost consumer asks for messages without a delay, whilst the high cost consumer will only see messages once delay passed).

What am I looking for?

3
  • Does this have to be solved in the queue? Can you simply tell your high-cost consumers not to consume messages if they haven't reached a certain age instead? Oct 7, 2022 at 6:26
  • @BenCottrell - many queue systems (SQS/RabbitMQ) lack a Peek operation so you have to then Pull/Requeue or Pull/Reject which would then risks message getting onto a DLQ.
    – Ryan
    Oct 7, 2022 at 7:32
  • RabbitMQ allows you to 'peek' by reading a message without consuming it; -- messages are only removed from a queue after the consumer sends an ack signal back to the broker (so if you pick up a message and send nack then it's available for another worker). SQS also has a similar acknowledgement mechanism. Oct 7, 2022 at 7:55

1 Answer 1

1

You can use RabbitMQ's Dead Letter Exchanges https://www.rabbitmq.com/dlx.html

  • set TTL on the message
  • put it into a queue (with x-dead-letter-exchange configured)
  • process the queue with low-cost consumers
  • if any message reaches the TTL (not processed by low-cost consumers) it gets moved to an other queue
  • process this other queue with high-cost consumers

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.