4

I monitor a variable in real-time where new value is generated every 1/25 second. Depending on the conditions, this variable may either be stable, i.e. fluctuate a little (+/-1%) around some value (for simplicity, I'll take real value of 40 from my recent experiments) or fluctuate widely from 0 up to 6x or 8x times bigger than stable value (200-370 for the example). My goal is to detect these stable periods.

Right now, I solve this problem like follows:

  1. measure mean and deviation of the incoming variable since the begginning of measurements;
  2. if at any given point the ratio of deviation/mean is greater than a threshold (0.15 right now) I consider that the variable has entered unstable period, otherwise - the variable is considered stable.

This approach generally worked, but not so good. Sometimes, detector fails to detect unstable periods. Here is the picture of recent experiment where ratio (in percentage) was less than 15%, detector considered period as a stable, however it can be clearly seen that variable becomes unstable towards the end of the graph (left axis is for the mean and "delta" - value of the variable; right axis for the deviation/mean ratio in percents):

enter image description here

As you can see, variable (blue line, delta) starts to fluctuate wildly at the end of the graph, however ratio deviation/mean stays under threshold (the maximum is around 8%) and doesn't trigger detector.

So I'm wondering, whether there are some existing oscillation detectors techniques/algorithms which can help to adjust the sensitivity of the detector? Currently, I'm thinking of using sliding average for the mean value and lowering the threshold up to 8%.


High-level system description

There is a producer which generates packets at constant rate, though it might get interrupted and inter-producing delay may not be exactly the one chosen (i.e. target is 25fps, but delay may vary between 30 and 50ms, very rarely it can be interrupted and delay may become twice as much). These packets are available on the network and cached. Consumer asks for these packets by issuing requests. It aims at exhausting the cache - by issuing large amount of requests. Those requests that ask for non-existent data, will become pending and answered as soon as corresponding packets are generated. Therefore, consumer will know that it has exhausted the cache once the data will arrive with "stable" period, i.e. the one that "roughly" corresponds to the target producer rate. As one can see, this inter-arrival delay is affected 1/ by producer (it's not exactly 40ms) and 2/ network disruptions.

Here's the sample data; there are bunch of additional parameters, but the real observed value is in "delta" column.

  • Yes, I can have a target stable value. Though it's preferred to not to rely on it. Do you think it's critical to know this value? The value is affected by network - it's the inter-arrival delay of received packets (I don't want to go deeper on this, as it'll require a lot of things to be explained). The packets are generated at constant rate and can be cached in the network. Thus, inter-arrival delay depends on how these packets are requested - cached packets mimics issuing pattern, new packets answers pending requests at producer's rate. Hope it's not confusing. – peetonn Sep 5 '15 at 19:20
  • Before I noticed your reply, I deleted my earlier comment. I was going to replace it with this: do you know the colour of the noise? Distribution? Is there an input signal to compare to so that we might establish a bounded input/bounded output relationship? Is the system linear? What sort of detector is generating the signal? Can you provide sample data to play with? – Timtro Sep 5 '15 at 19:24
  • My concern is that you might not have guarantees that the system is stable for long enough for you to converge on any notion of stable behaviour. But we can certainly work around it, probably at the cost of sensitivity. – Timtro Sep 5 '15 at 19:26
  • I suppose it's a white noise (I might be wrong). I'll try my best describing system on high-level, please see the update to the answer. – peetonn Sep 5 '15 at 19:44
  • Okay. I'll post some ideas. Since there are a lot of unknowns, we may need to play around to see what works. You okay with that? – Timtro Sep 5 '15 at 19:45
1

I tested a solution that keeps a running average of the previous N samples, and N samples before that. This has the effect of smoothing the noise out, but you can compare the differences between the two averages to determine if any significant change has occurred between the two.

If this solution doesn't work, I have other ideas, but this is simple and seems to work well with the sample data you posted.

I did a mock up in Python. I'm not sure of your comfort level with that, so I didn't make it very pythonic, so that it could be easily transcribed to whatever language you're using.

import numpy as np
from collections import deque
import pylab as plt

N = 25
windowA=deque(maxlen=N)
windowB=deque(maxlen=N)

avgA = []
avgB = []

with open('stab-estimator - Sheet1.tsv', 'r') as f:
  f.readline() # skip header
  for line in f:
    delta = float( line.split()[1] )
    windowB.append(delta)
    windowA.append(windowB[0])
    # I'll store the averages so we can plot them, but you can just use
    # avgA/avgB as a metric for stability, without storing a history.
    avgA.append(np.sum(windowA)/len(windowA))
    avgB.append(np.sum(windowB)/len(windowB))


# Plot the result:
avgA = np.array(avgA)
avgB = np.array(avgB)
plt.figure()
# plt.plot(avgA-avgB)
# Or equally telling:
plt.plot(avgA/avgB)
plt.show()

The plot of that ratio looks like this: enter image description here

As you can see, it presents a good metric for your observed instability. You could easily set a threshold of 10% and catch that instability at the end. You may also want to tune N. The size of N will be a compromise between smoothing and sensitivity to changes. If N is too large, then the signal will have to grow more quickly to overcome the large averaging window.

Also, you will have to mute the alarm at the beginning while the averages converge to running values.

HTH.

  • thanks! i'm fine with python (don't mind if you can share the code). will check this tonight! – peetonn Sep 6 '15 at 2:01
  • I tried your solution on real data, it seems to perform better. I'll mark your post as an answer. Thanks! – peetonn Sep 9 '15 at 17:41
  • @peetonn If you need something better, let me know. I'm on vacation now, but we can work on it next week. – Timtro Sep 11 '15 at 1:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.