2

Here is my exact scenario.

  • I must make requests to a third party service
    • The service takes ~15 seconds to respond
    • It also has no webhooks or any call back mechanisms
    • These requests are not made constantly throughout the day, but in chunks
  • The requests to the third party service are made from a consumer that is pulling from a queue
  • using mass transit default impl & rabbitmq
  • All of the messages the product generates are currently put into one queue. The same queue with the calls to the third party service
  • There are currently 3 consumers, running 4 worker threads each for a total of 12 worker threads. This is the current maximum number of concurrent messages that can be handled at once
  • Throughout the day the message queue will receive ~100 of these messages that will back up the queue. All 12 worker threads will process the ~15s calls and the queue will begin to backup. This results in extreme loss in functionality. Delayed emails, etc. It normally continues to grow the queue for ~30 minutes until it catches up
  • When this occurs the consumers are running at 0% CPU because they are all in thread waits.

The question I have is how do I setup my message queue and consumers so that they don't backup. I am trying to prevent loss of functionality of the site for those 30 minute time blocks. Anything goes.

Recap:

  • 1 queue
  • ~15 seconds tasks running at 0% cpu
  • 12 consumer worker threads
  • Queue backs up and takes ~30 minutes to catch up
  • using mass transit default impl & rabbitmq
  • So the problem is that some of the messages (not all, probably not even most) in your queue require blocking calls to the 3rd party service? – Dan Pichelman Jul 24 '15 at 19:29
  • Yes that sounds about right. "Require" is a sliding term. Right now, they are blocking calls. – BradLaney Jul 24 '15 at 19:33
  • A couple of ideas - a) put those messages in a separate queue and dedicate threads to them, or b) limit the number of threads that can do these tasks – Dan Pichelman Jul 24 '15 at 19:36
  • Either way you have consumer worker threads that are reserved for sitting around doing nothing all day. And worker threads are limited. – BradLaney Jul 24 '15 at 20:11
  • True. I wonder - is there a way for a thread to look & see that there are 9 or 10 blocked threads and therefore it shouldn't accept blocking work? That gets messy really fast due to concurrency issues. – Dan Pichelman Jul 24 '15 at 20:14
2

There are a couple different ways you might deal with long running blocking tasks:

  1. Increase the number of workers. As long as CPU utilization in staying low, additional workers will allow you to more quickly dispatch these long running service calls. More workers only becomes problematic at the point that some resource is under contention (cpu, memory, network, disk, etc). You mention in a comment that "worker threads are limited", but in an application with very long running blocking tasks it is not uncommon to throw 100s if not 1000s of worker threads at the problem.

  2. If the problem is that long running tasks are starving other work sent to the same queue, you can prioritize the messages. This way workers will always pickup the next highest priority task. However, if your queue depth keeps growing because you cannot process messages fast enough you'll need more workers.

  3. Another approach to reducing starvation would be to send the messages to different queues which have their own workers. This will offer the lowest latency for your non-blocking tasks and make it easier to balance resource usage when tuning the number of workers.

  • 2
    If only there were a way to make these calls asynchronously... – Robert Harvey Jul 24 '15 at 22:45
  • @RobertHarvey Yes, non-blocking models make this a lot easier. But calling the service with an async API is half the battle, the messaging client also needs to support delaying message acceptance. It looks like Mass Transit 3 (not out yet) will make this easy. – Chris Pitman Jul 25 '15 at 4:46
  • @ChrisPitman 1. These calls are sporadic and only show up every hour or so. If I increase the number of workers, then commonplace tasks will then raise CPU. Our workers are at the current threshold for general operation. 2. If I have the workers pick up the non-running tasks first, there is high probability the long-running tasks won't ever run. Right now the only reason they run is because ONLY they run and it takes half the time until the next set of messages trickles in. 3. I was thinking of sending them to different workers, but now I have a whole server dedicated to running at 0% CPU. – BradLaney Jul 27 '15 at 15:21
  • @BradLaney You can run multiple types of workers on a single host. Over-subscription of resources is not always bad, especially when you have intermittent loads like this. – Chris Pitman Jul 27 '15 at 20:04
  • I just want to solidify my idea of what you are saying. Are you suggesting we create another service (consumer) that is bound to a different queue and runs it's own worker threads on the same host as the original worker queue? And you believe this constant context switching of ~CPU*2 threads is good practice for normal behavior just to handle a 1% use case? – BradLaney Jul 28 '15 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.