Think about your KPIs.
I bet they're related to 95th percentile latency.
That is, the reason for scaling out,
for sending more dollars to Amazon hosting,
is to keep customers happy.
So define, and measure, some metrics of interest to the business.
Let's examine those four tasks at midnight.
Your customers are mostly sleeping, right?
You could scale in to a smaller number of tasks and
no one would notice. It literally doesn't matter.
So spend fewer dollars on compute resources.
Now let's look at lunch time behavior.
A typical model for customer arrivals is a
It is memoryless -- customer arrivals are
independent of one another.
We wish to estimate the arrival intensity:
Lambda will change with time-of-day
and with calendar season.
Before the rush-hour customers converge
on your site simultaneously, you want to
anticipate the intensity of their traffic
and pre-spawn an appropriate number of
servers to accommodate their requests.
Now, down to brass tacks.
At some timestamp
t a new customer request arrives.
k servers running, maybe the default of 4.
The most important statistic for the customer is:
What is the probability that at least one server is idle?
That is, that an idle server will immediately pick up
Use a Poisson model,
or recent server statistics,
to estimate whether that new customer's service
time will be worse than the 95th percentile delay,
or whatever relevant stats your
If the delay is too high, suggesting
a good chance of new request being met
with "all servers are busy", then spawn
a new server. Conversely, if the probability
of delayed service is low ("many servers are idle"),
then kill off a server.
For stability in this
you will want a bit of
High / low probabilities are one approach.
Simply waiting an interval between control actions,
like sixty seconds, is another approach.
λ lambda estimates are not readily available,
a proxy like recent queue-depth could readily be a stand-in.
Exponential smoothing of such a measure is straightforward.
Again, high / low hysteresis limits will help avoid
scenarios where alternate control intervals ask
for +1 server, -1, +1, -1, ....