1

I have a service which is triggered by messages from three queues (which are populated by different topics), and writes the processed results to another system (say DB service) synchronously. The incoming messages are of type insertion, update and deletion of data for the customers of our system. So ordering of the handling of the messages is important, i.e. if we get messages like:

Timestamp 1: (Queue1, Update single entity message)
Timestamp 2: (Queue2: Delete all data message)

And if the services processes queue1 message after processing queue2 message, then we would reach an undesirable state, since we would be storing data for the customer even though the latest activity from them was to ask for deleting all of their data.

To complicate matters, the service where we write our results throttles our calls, allowing only a fixed number of transactions by a customer per second. This leads our service to regularly send incoming messages to the dlq after five failures, which further messes up the order in which we write our results to the DB service.

I'm trying to solve the problem of the messages frequently ending up in the dlq, and considering the following approaches:

Approach 1: If a message is throttled, sleep for some time and retry writing again. I'd need to increase the retry count for the synchronous calls and increase the timeout after which the service makes the message available in the queue again.

Pros:

  • Good chance of processing a message once the service handles it.

Cons:

  • Messes up the ordering of the messages. Though this would still be better than the current scenario of message being in the dlq and being manually redriven after a couple of days.
  • If threads are waiting, holding a message, it would slow down the processing of the incoming events, and I would need to scale up the system to have a similar throughput rate.
  • If the DB service is operating a fraction of its capacity, multiple retries may brown out the service.

Approach 2: Token bucket: When the service wants to read a message, it needs to have a token available. Once it gets the token, reads, processes the messages, and writes to the DB service. Once it is done successfully writing to the DB service, it puts back the token. Since there are a fixed number of tokens, lining up with the throttle rate of the DB service, the rate of the incoming messages is controlled, which eventually controls the output rate of the service.

Pros:

  • Good chance of processing a message once the service handles it.

Cons:

  • Messes up the ordering of messages, for a horizontally scaled up service.
  • Additional complexity for maintaining the token bucket logic.
  • Need to scale up to match the current throughput rate.

  1. What other approaches can be used to limit the output rate? Is there a common design pattern for these kinds of services to overcome such issues?
  2. Where can I read up more about these sort of problems and approaches to solve them?
  3. Is the architecture of my system fundamentally flawed (processing write and delete which need ordering in a horizontally scaled queue based system), and should I work on fixing the architecture instead of trying out ways to solve the side effects of the architectural issues?
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.