2

I'm storing some temperature, voltage and current data every 10 seconds to a log file. This file is supposed to be retrieved later so we can read the data and display it. Time requirement is 6 month worth of logs and the space requirement is under 6 MiB.

The protocol I came up with looks something like a fixed record of 6 bytes:

  • PID 2 bytes
  • Data 4 bytes

if PID is a timestamp data is either date (4 bytes) or time (4 bytes). So I put one timestamp and whatever measurement that follows at that time.

I tried storing a timestamp followed by the three readings linearly with fixed records but quickly ran out of space. Then I tried only tracking changes of values, storing the last value and the current value if there was a change of data.

Since the timeline is tree structured (YYMMDDhhmmss) I was thinking of sorting the data in a tree structure sorted on a timeline instead, making the record size more packed, but how would you go about storing this data for easy request based retrieval later if it is stored in a tree?

/edit related questions here: How to store data that is recorded with different frequency? answer suggestions are similar to what I have come up with

5
  • Wait, are you sampling regularly (every 10 secs) or not? – candied_orange Mar 14 '18 at 23:33
  • Usually, you must pick two between interval, data size and storage. Trying to put limits on all three makes things way, way harder... – T. Sar Mar 14 '18 at 23:42
  • @CandiedOrange sorry, forgot to add that some values are to be logged at 60 sec intervals, so it varies a little but I figured it could be rendered an implementation detail with the system I came up with, so question remains largely the same I hope – Fred Mar 14 '18 at 23:47
  • Ok but everything that is logged is logged at regular intervals? – candied_orange Mar 14 '18 at 23:48
  • @CandiedOrange That's correct. With 1 second granularity as the smallest unit (so that makes 6 logs per minute for all three of them, and every minute there is two extra readings) – Fred Mar 14 '18 at 23:51
6

Since you said space was at a premium, so much that your worried about timestamp format, and since your data payload is small, and you're logging at fixed intervals, I'd make the timestamps into a header that is not added every time. (You'll have to imagine these timestamps in binary.)

So rather than

<timestamp> <data>

2018-03-14T23:03:00Z 01 23 45 67
2018-03-14T23:03:10Z 01 23 45 67
2018-03-14T23:03:20Z 01 23 45 67
2018-03-14T23:03:30Z 01 23 45 67
2018-03-14T23:03:40Z 01 23 45 67
2018-03-14T23:03:50Z 01 23 45 67

You could have

<timestamp> <data> <data> <data> <data> <data> <data>

2018-03-14T23:03:00Z
01 23 45 67 01 23 45 67 01 23 45 67 01 23 45 67 01 23 45 67 01 23 45 67

2018-03-14T23:04:00Z
01 23 45 67 01 23 45 67 01 23 45 67 01 23 45 67 01 23 45 67 01 23 45 67

Where each entry is data gathered at a different time. The time stamp only shows when the first one is entered. Each are gathered at a time evenly divided over the time between timestamps.

This trick only works when what you're logging happens at fixed times. So it would require you to put things that are logged at different rates in their own files. This assumes that everything is being logged at a predictable time and that you will not fail to log something when it's time to log data. You may need a flag value to represent that data isn't available.

The choice to use 6 data samples per timestamp is arbitrary. The more samples per timestamp the more compact the file will be and the bigger the hassle will be to look up particular times. At the extreme end you could do away with the timestamp altogether and let the files creation and update dates take care of marking time.

Of course logs can be rolled over and compressed so that this is all unnecessary. It's really about which kind of hassle you're willing to put up with.

2

The usual solution is to store only differences, use a variable byte size encoding, and follow that with a fast compression algorithm, assuming you can keep two files, a smaller uncompressed portion, and a larger portion consisting of compressed blocks. When the uncompressed portion reaches a certain size, you compress it and append it to the compressed file.

Start with an integer like representation of your data (it can be 16, 32, 64, any size). Keep as state the last value for this data. Xor these two to obtain the delta. Then record the difference using something like 7bit encoding, where the top bit indicates if there is another byte to be read, storing from LSB to MSB.

I have used L74 for fast data compression and it worked very well. You can compress in fixed sized blocks.

Finally when you need to retrieve the data, decompress in memory, and you can index easily.

1

Do you have a handle on how much the data values will vary from one reading to the next? If so, you may be able to reduce the size of the data, at the cost of not using a fixed-size record. Your current criteria appear to be:

6 bytes * 6 readings/min * 60 min/hr * 24 hr/day * 30 day/mo * 6 mo = 9,331,200 bytes

but you want to fit it into 6 MiB, which is 6,291,456 bytes, so you need a 3:2 reduction in size.

If you know that temperature, for example, won't vary by more than 5 degree C, you could take the first reading and store it, then store the delta in only 4 bits (-8 to +7). Likewise with voltage, and current. It does make finding a particular reading more difficult. You could pick some interval at which you save the full reading to make that easier (once per hour, or once per day, for example). If all of your readings were just temperature, and you did this, you'd have 6 bytes for the first reading and 2.5 bytes for each subsequent reading until the end of the time interval. Let's say you do a full write once per hour. It would come out to:

(6 bytes + (2.5 bytes * 6 readings/min * 60 min/hr)) * 24 hr/day * 30 day/mo * 6 mo = 3,913,920 bytes

You could combine that with only outputting the time stamp once per hour, to make it even smaller, as most of the data would become .5 bytes instead of 2.5 bytes. (Again, this is if it were only temperature data and the deltas were bounded.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.