How to auto-scale a service based on requests count and internal processes

Question

I have a service which is used to register tasks. These tasks are asynchronous and executed in background using a state management engine.

The state management engine is running inside the service and it is executing 4 concurrent tasks at a time. I have an algorithm to auto-scale the service based on the number of tasks that are waiting in the queue to be executed and I want to write an algorithm to auto-scale the service based on the number of requests to register the tasks such that these algorithm doesn't interfere in each other's scaling.

For example: If the service has scaled-out based on the number of tasks present in the queue then based on the requests to register the tasks, it should not scale-in.

Is there any real world example where this kind of auto-scaling is happening? What would be the best possible way to achieve this kind of auto-scaling that is dependent on two different scenarios?

Any reference or guidance would help.

Answer 1

Think about your KPIs. I bet they're related to 95th percentile latency. That is, the reason for scaling out, for sending more dollars to Amazon hosting, is to keep customers happy.

So define, and measure, some metrics of interest to the business.

Let's examine those four tasks at midnight. Your customers are mostly sleeping, right? You could scale in to a smaller number of tasks and no one would notice. It literally doesn't matter. So spend fewer dollars on compute resources.

Now let's look at lunch time behavior. A typical model for customer arrivals is a Poisson process. It is memoryless -- customer arrivals are independent of one another. We wish to estimate the arrival intensity: λ lambda.

Lambda will change with time-of-day and with calendar season. Before the rush-hour customers converge on your site simultaneously, you want to anticipate the intensity of their traffic and pre-spawn an appropriate number of servers to accommodate their requests.

Now, down to brass tacks. At some timestamp t a new customer request arrives. We have k servers running, maybe the default of 4. The most important statistic for the customer is: What is the probability that at least one server is idle? That is, that an idle server will immediately pick up the request.

Use a Poisson model, or recent server statistics, to estimate whether that new customer's service time will be worse than the 95th percentile delay, or whatever relevant stats your SLA calls for. If the delay is too high, suggesting a good chance of new request being met with "all servers are busy", then spawn a new server. Conversely, if the probability of delayed service is low ("many servers are idle"), then kill off a server.

For stability in this control process you will want a bit of hysteresis. High / low probabilities are one approach. Simply waiting an interval between control actions, like sixty seconds, is another approach. If λ lambda estimates are not readily available, a proxy like recent queue-depth could readily be a stand-in. Exponential smoothing of such a measure is straightforward. Again, high / low hysteresis limits will help avoid scenarios where alternate control intervals ask for +1 server, -1, +1, -1, ....