Which datastructure to use for temporary storage of intermittent and asynchronous sensor data?

Question

Here an outline of the setup:

Input data:
I have several sources for sensor input data (GPS, and a bunch of other sensor data from different sources).

• All of those individual inputs consist of a timestamp, the name of the input and the value.
• There is no defined list of inputs, but new inputs can appear whose name we didn’t know before. Just as well inputs can disappear again.
• The update rate of the Inputs is also neither predifined nor fixed. It could be periodic or sporadic, but we don’t know its recurrence before and this is also subject to change.

My goal:
I want to report the current state of Inputs (only the newest value) in a fixed recurrence, e.g. every second, via MQTT to a Server.
There is no need to store the input data after sending the MQTT message. When a specific sensor input is not available for several seconds, it shall not be part of the MQTT message anymore.

My problem:
How do I store the input data before sending the current “snapshot” in the fixed recurrence to the MQTT server?

My current solution (?!ugly hack?!):

• Everytime a new sensor input value appears, I write the timestamp and the value in a file with the name of the sensor input in a single folder which is used for all inputs.
• If the file already existed, I simply overwrite it, as only the newest value of the sensor is important.
• To not have too old sensor input data lying around, I remove files that were created more than a defined number of seconds ago.

• Example: Sensor Input GPS latitude is stored in file latitude.dat, which has one line containing the timestamp and the value of latitude:

  pi@raspberrypi ~ $ cat latitude.dat
1534448235 46.2439877495

The solution works and is flexible enough to handle this task, but I am not at all happy about it. I thought about using a SQL-database, but I somehow think due to the nature of the inputs, it’s also not optimally suited. Does anybody have a good idea, what other data structure I could use for the storage of the input data?

EDIT: The sensor data does not need to survive a power failure.

Answer 1

If you really only need to report the latest value and nothing else, i.e. no historical data, then all you need is a simple, non-hierarchical, key-value store with the sensor name as the key and the sensor value as the value, and just ignore the timestamps.

There is no need to distinguish between existing and new sensors. If a sensor reading comes in, you just the key to the value. If it is an existing sensor, then this will overwrite the existing value, if it is a new sensor, this will create a new key-value pair.

Note, this makes certain assumptions:

• the timestamps are irrelevant
• sensor readings will not arrive out of order
• reads and writes are atomic OR a single out of date sensor reading doesn't matter
• sensor reports don't need to be consistent across all sensors

If sensor readings can arrive out of order, you also need to store the timestamp, and compare the timestamp in the reading to the one stored in the database to make sure that the newly-arrived reading is actually newer.

Obviously, if timestamps are part of your sensor report, then you need to store them as well.

If reads and writes are not atomic, then it can happen that the reporting part of the system reads a sensor value just as the sensor reading part is writing it. In this case, you need to lock that particular key-value pair while reading / writing, or risk sending out corrupted or at least out-of-date sensor values.

If the sensor report needs to be consistent, i.e. the values of all sensors need to be the latest one at a particular time stamp, then you need to lock the entire database when sending a report.

Whether or not you do this in-memory, on-disk, eventually-on-disk and what your consistency models are, depends on your requirements. Does the database need to survive a power failure? Or is it okay to send out a couple of empty reports until all sensors have sent new updates?