I'd need some feedback on how to approach the design of a function that highlights parts of a time series chart. The chart shows the response time of an application, in particular the 90th percentile, over time. Each bar on the chart is one minute. I want to see anomalies, determined based on thresholds I define.
I expect I will be able to obtain a threshold, maybe based on historical data, of what's the expected value of the 90th percentile (as in, "what's normal"). Same goes for another threshold for the extreme values of the 90th percentile, also based on historical data. Let's call these thresholds T1 and T2, respectively.
What I want to do: I want to be able to enhance the chart in such a way that I can see "unusual deviations" of the 90th percentile. This will be in particular useful to find issues we saw after releasing a new software version of the application - check if there are extreme, prolonged spikes, if there are consistent, but not extreme, degradation of performance, etc.
Now, to the key point, what is unusual?
- Consecutively exceeding the lower threshold T1. The 90th percentile has been above the lower threshold, for say X=10 minutes in a row.
- a) Extreme peaks, exceeding higher threshold T2. The 90th percentile has been exceeding the extreme values for say X=5 minutes in a row. b) I think to make the extreme values measure more useful, there needs to be a notion of "close to threshold T2", since I expect T2 to be probably twice as high as the lower threshold T1. So, exceeding threshold T2 might be already too late/extreme.
- To make these measures somewhat robust, we should ignore intermittent dips, so the sequence of values extreme, extreme, normal, extreme would still be considered a single series of extreme values.
- After a deviation is over, the highlighting, presumably based on a running score, needs to return back to normal quickly. I don't want to see highlighted data points for an issue that happened 1 hour ago. Think of exponential drop-off.
Let's look at a made-up response time chart:
W1 would be an example of scenario 1. and W2 would be an example of scenario W2. Note that we don't care if the values are below the T1 threshold, or if there are single, extreme values. At least for now.
What I'd like to get out after my model colors the graph:
Based on the second chart, I'd know that I need to look at the two red timeranges and have a closer look what happened there.
I don't expect someone to provide me the finished code, but I could use a few pointers:
- Examples of existing implementations. I found fancy machine learning anomaly detections, but I'm not interested in that. What I'm looking for is a basic model with tunable parameters such as thresholds and timeframe length. The model should not be too involved and easily explainable, such that I can implement it myself based on basic stats functions.
- Many examples I found are based on normally distributed data. The percentile distributions I work with are for sure not normally distributed. They maybe follow loosely a power law distribution, but I'd like to avoid assuming a certain distribution - is that doable?
- Any other input you have if you worked on a system like this before.
I'm OK with the basics of math and some stats.