1

Here an outline of the setup:

Input data:
I have several sources for sensor input data (GPS, and a bunch of other sensor data from different sources).

  • All of those individual inputs consist of a timestamp, the name of the input and the value.
  • There is no defined list of inputs, but new inputs can appear whose name we didn’t know before. Just as well inputs can disappear again.
  • The update rate of the Inputs is also neither predifined nor fixed. It could be periodic or sporadic, but we don’t know its recurrence before and this is also subject to change.

My goal:
I want to report the current state of Inputs (only the newest value) in a fixed recurrence, e.g. every second, via MQTT to a Server.
There is no need to store the input data after sending the MQTT message. When a specific sensor input is not available for several seconds, it shall not be part of the MQTT message anymore.

My problem:
How do I store the input data before sending the current “snapshot” in the fixed recurrence to the MQTT server?

My current solution (?!ugly hack?!):

  • Everytime a new sensor input value appears, I write the timestamp and the value in a file with the name of the sensor input in a single folder which is used for all inputs.
  • If the file already existed, I simply overwrite it, as only the newest value of the sensor is important.
  • To not have too old sensor input data lying around, I remove files that were created more than a defined number of seconds ago.
  • Example: Sensor Input GPS latitude is stored in file latitude.dat, which has one line containing the timestamp and the value of latitude:

    pi@raspberrypi ~ $ cat latitude.dat
    1534448235 46.2439877495

The solution works and is flexible enough to handle this task, but I am not at all happy about it. I thought about using a SQL-database, but I somehow think due to the nature of the inputs, it’s also not optimally suited. Does anybody have a good idea, what other data structure I could use for the storage of the input data?

EDIT: The sensor data does not need to survive a power failure.

  • Does your application need to survive a power failure? Why bother writing this data to a file? Why not keep it in memory? – Dan Pichelman Aug 16 '18 at 20:03
  • No Need to survive a power failure. I will be happy to not rely on writing to files anymore. – oh.dae.su Aug 16 '18 at 20:23
2

If you really only need to report the latest value and nothing else, i.e. no historical data, then all you need is a simple, non-hierarchical, key-value store with the sensor name as the key and the sensor value as the value, and just ignore the timestamps.

There is no need to distinguish between existing and new sensors. If a sensor reading comes in, you just the key to the value. If it is an existing sensor, then this will overwrite the existing value, if it is a new sensor, this will create a new key-value pair.

Note, this makes certain assumptions:

  • the timestamps are irrelevant
  • sensor readings will not arrive out of order
  • reads and writes are atomic OR a single out of date sensor reading doesn't matter
  • sensor reports don't need to be consistent across all sensors

If sensor readings can arrive out of order, you also need to store the timestamp, and compare the timestamp in the reading to the one stored in the database to make sure that the newly-arrived reading is actually newer.

Obviously, if timestamps are part of your sensor report, then you need to store them as well.

If reads and writes are not atomic, then it can happen that the reporting part of the system reads a sensor value just as the sensor reading part is writing it. In this case, you need to lock that particular key-value pair while reading / writing, or risk sending out corrupted or at least out-of-date sensor values.

If the sensor report needs to be consistent, i.e. the values of all sensors need to be the latest one at a particular time stamp, then you need to lock the entire database when sending a report.

Whether or not you do this in-memory, on-disk, eventually-on-disk and what your consistency models are, depends on your requirements. Does the database need to survive a power failure? Or is it okay to send out a couple of empty reports until all sensors have sent new updates?

  • Thanks a lot for your answer! Regarding your questions: 1) The timestamp shall be stored and sent to the MQTT Server. 2) Sensor readings arrive in order. 3) reads and writes are atomic. 4) The data does not need to survice power off. One big requirement is that when a sensor Input is not available for several seconds, it shall not be reported anymore via the MQTT protocol. How can I achieve something like this with a key-value store without the overhead of simply going through the whole store and looking for outdated values? – oh.dae.su Aug 16 '18 at 20:21
  • Ok I think I got the idea of also how to get rid of the outdated values. Everytime I send the MQTT message I have to go through the whole key-value store anyhow. I can then simply delete the outdated key-value pairs in this step. – oh.dae.su Aug 16 '18 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.