I've been asked to develop a "telemetry" application that records data generated by a hardware device, which I read every 100ms.
There are approx 250 data points (32-bit values), but only a subset of these (say 30-50) will be returned by each 100ms read, so it will take several reads before I've got all 250 data points. Out of the data point values read, only a small number are likely to have changed since those same data points were last read - some change every 100ms, others perhaps every few seconds, while others very infrequently (or even never).
The main requirements for the system are:-
- The ability to plot one or more data points over time, at 100ms resolution
- The ability to view the values of all 250 data points at a particular point in time
- The application needs to store the last 60 hours worth of data
I'm looking for some thoughts on how best to tackle this, particularly how to store the data in a way that supports the required reporting. So far I'm veering towards this idea:-
I maintain a "dictionary" of all 250 data points, updating the values after each read. Every 100ms I would write all 250 values to my data store. It's easy to implement and easy to query the data to satisfy the requirements. The downside is that it's not a particularly efficient use of storage (but hey - disk space is cheap nowadays).
Alternatively, ever 100ms, I could store only values that have changed since they were last read, however this becomes more difficult to implement, especially when it comes to querying. E.g. for requirement #2, I would have to construct a "snapshot" by traversing back through the data from the required point in time, looking for each data point's last stored value.
Now I've written all this out I think I've answered my own question. The "write everything" approach seems to be a no-brainer. However I've had no experience of this kind of mass data storage and analysis so I would be interested to hear what others think.