I have a stream of integer data and want to perform some statistical analysis on it. I want to calculate the mean and the standard deviation of it. So far it isn't hard, but keep in mind that I am talking about streams of data, I'd prefer not to store all of the data. There exists both an algorithm for the mean and the deviation to keep the stored data at a minimum - I'd refer to Wikipedia in this matter.
But the problem now is that some of the data will be completely absurd in regards to the rest of the data. For example I will receive
1 2 2134 7 -2 14 // 2134 is out of line and junk, don't calculate with it
I know in what ranges my values will likely be, but only relative to their average value. So I'd like to know if there is a good approach to tackle these kind of noises.
It is even more annoying, as the junk data will most likely be the first to arrive and not in line with the mean value of the rest of the sequence so I can't precalculate the mean value. An example would be something like
1111 1564 13 1645 12 -4 37 90 ...
The junk makes
< 5% of the data, so at least there isn't much noise, but I have to keep it out of my calculation.
To distinguish between real and junk data, I know that the real data lies in a bounded interval around the mean value - scaled to the example above, it would be something like +-200. "Groups" of junk data can behave in the same way, only that their mean value differs by at least 1 times the length of the interval from the mean value of the real data.
At the beginning, without any data, I have no idea what the mean value of the real data might be.
I can replay a limited amount/small fraction of data only by first saving it to memory.
Is there a good algorithm that occupies as little as possible space and works when applied iteratively to a sequence?