We have a number of systems that are based on the competing consumers pattern. We currently monitor the delay between an item of work being queued and processing finishing, however, as throughput increases, this metric will only increase over the average when the system is already above its maximum capacity - the point at which the volume of incoming requests is greater than the systems ability to process them. At this point, it is too late, and processing times will continue to increase until the volume of work decreases, or we are able to add capacity.

I would like to implement a monitoring system that tells us how close we are to being over capacity before this happens. Are there any standard patterns for this sort of monitoring?

e.g. my current thoughts are that I could measure the total time spent processing requests as a percentage of the theoretical maximum, e.g. if in a 60 second period a total of 450 seconds are spend processing requests over a maximum of 10 threads, this indicates that we are at 450 / 600 = 75% of theoretical maximum capacity. Is this approach sensible?


Well it sounds like straightforwardly the metric you're measuring is in fact the wrong one and isn't useful to your problem.

There are several metrics you could consider to measure, with different strengths and weaknesses.

  • "time to pickup". That is, how long a message spends sitting in the queue before it's accepted for work by a consumer. This metric will trend upwards before you're overloaded, it's like a metric of backpressure from your consumers. It trends upwards as they become busy working on tasks that they can't take new ones off.
  • "Clear time". If you do this sort of thing constantly, you should be able to look at a given task on the queue and take a rough guess at how long it will take to process. Sum that for your available tasks, and you can compute a "clear time". When actual clear time deviates from predicted clear time, you're starting to get in trouble. When you're really badly in trouble, clear time will be double or triple predicted. Clear time is basically the amount of time it takes for your system to run dry of tasks under the assumption that all message production ceased right now.
  • Why is the metric I describe not useful? – Justin Sep 11 '17 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.