How do I handle long running third party calls from backing up my message queue?

Question

Here is my exact scenario.

• I must make requests to a third party service
  • The service takes ~15 seconds to respond
  • It also has no webhooks or any call back mechanisms
  • These requests are not made constantly throughout the day, but in chunks
• The requests to the third party service are made from a consumer that is pulling from a queue
• using mass transit default impl & rabbitmq
• All of the messages the product generates are currently put into one queue. The same queue with the calls to the third party service
• There are currently 3 consumers, running 4 worker threads each for a total of 12 worker threads. This is the current maximum number of concurrent messages that can be handled at once
• Throughout the day the message queue will receive ~100 of these messages that will back up the queue. All 12 worker threads will process the ~15s calls and the queue will begin to backup. This results in extreme loss in functionality. Delayed emails, etc. It normally continues to grow the queue for ~30 minutes until it catches up
• When this occurs the consumers are running at 0% CPU because they are all in thread waits.

The question I have is how do I setup my message queue and consumers so that they don't backup. I am trying to prevent loss of functionality of the site for those 30 minute time blocks. Anything goes.

Recap:

• 1 queue
• ~15 seconds tasks running at 0% cpu
• 12 consumer worker threads
• Queue backs up and takes ~30 minutes to catch up
• using mass transit default impl & rabbitmq

Answer 1

There are a couple different ways you might deal with long running blocking tasks:

1. Increase the number of workers. As long as CPU utilization in staying low, additional workers will allow you to more quickly dispatch these long running service calls. More workers only becomes problematic at the point that some resource is under contention (cpu, memory, network, disk, etc). You mention in a comment that "worker threads are limited", but in an application with very long running blocking tasks it is not uncommon to throw 100s if not 1000s of worker threads at the problem.

2. If the problem is that long running tasks are starving other work sent to the same queue, you can prioritize the messages. This way workers will always pickup the next highest priority task. However, if your queue depth keeps growing because you cannot process messages fast enough you'll need more workers.

3. Another approach to reducing starvation would be to send the messages to different queues which have their own workers. This will offer the lowest latency for your non-blocking tasks and make it easier to balance resource usage when tuning the number of workers.