We have a number of systems that are based on the competing consumers pattern. We currently monitor the delay between an item of work being queued and processing finishing, however, as throughput increases, this metric will only increase over the average when the system is already above its maximum capacity - the point at which the volume of incoming requests is greater than the systems ability to process them. At this point, it is too late, and processing times will continue to increase until the volume of work decreases, or we are able to add capacity.
I would like to implement a monitoring system that tells us how close we are to being over capacity before this happens. Are there any standard patterns for this sort of monitoring?
e.g. my current thoughts are that I could measure the total time spent processing requests as a percentage of the theoretical maximum, e.g. if in a 60 second period a total of 450 seconds are spend processing requests over a maximum of 10 threads, this indicates that we are at 450 / 600 = 75% of theoretical maximum capacity. Is this approach sensible?