2

I have the following case:

We register data from sensors. Let's say that it is a value reported every second, for one hour a day. So we have over 3k values. These values are processed to try and eliminate sensors' faulty behaviors. At the end, there are some performance indicators calculated based on the processed set of data.

At this moment each step of the pre-processing stage is handled by a dedicated class. They act like commands: eliminate unrealistic readings, fill in missing readings, etc. The commands are loaded and executed in an order defined by the configuration file. The final step (calculating performance indicators and storing them in the DB) is like a projection (as far as I understand the concept).

I would like to start storing the steps of the pre-processing stage, so they can be replayed (to check how each of them affects the projected performance indicators, or to fine tune them). But each command changes a lot (if not all) data. So it will be quite expensive to store each change like it should be done with Event Sourcing. So my idea was to store the commands instead (perhaps with some parameters specific to each command).

Is it a good approach? Or should I still do Event Sourcing?

0

I believe you're on the right track - I'll quickly outline a solution, but if I understand you correctly that's more or less what you've suggested.

Your application consists of three steps:

  • Get a stream of sensor readings
  • Apply a number of transformations to the stream
  • Calculate various metrics based on the stream

Note that everything is ultimately derived from the raw sensor data. Therefore, I would suggest that you store this data.

Now you can apply your transformations to the data stream, but I see no reason to store the result of each individual transformation. In fact, I would not even store the final result of all transformations. Or the performance metrics, for that matter.

Instead, whenever you want to display the performance indicators, you load (or generate) the appropriate configuration and re-apply the transformations to your raw data. Then you can calculate the performance metrics and display them.

This way, you can choose to apply any transformation you need. You can add new transformations in the future, reorder transformations, remove them and determine the effect of each transformation on the result.

If the amount of data you handle increases, or your transformations are quite complex, this approach might cause a performance problem. At that point, you will have to cache some of the results. The basic principle, however, should not change: you can calculate all metrics from the raw data - for some of them, you just don't.

I recommend you start simple and only add caching when you really have to - and only to the extent you actually need it. The monetary cost of storing the additional data is almost certainly negligible. The cost in flexibility, and the danger of inconsistencies in you data set, on the other hand, is quite real.

  • Hi, thank you for your answer @doubleYou. My idea to store the order of transformation comes from the fact that I want to cache the metrics (cause they are used later by some complex SQL queries). So I want to be able to see what resulted in metrics being calculated in one or another way. – george007 Mar 17 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.