We're building a web application to monitor industrial facilities, and we need to add a module to notify the users when certain things happen in the system.
We get data (lots of data) from the facility machines as 10 minutes data series (a value each 10 minutes, like machine's components temperatures, output performance, power consumption, etc), and we store the data in a cassandra DB. The framework used is Django.
These are the kind of things that we need to notify (and we need to emit the notification when the event happens, so no batch processing):
When a machine has been under 90% of its theoretical performance for 60 minutes in a row (i.e. 6 rows of 10-min data), but not if the machine has been manually stopped.
When the whole facility has been under 95% of it theoretical performance for two hours in a row.
When a machine is manually stopped.
When the monthly production reaches 50% of the expected.
The details of the notification (percentages, time spans, etc) should be configurable per user.
We are looking for a design as generic as posible, because the list of available notifications will grow in the future. Has somebody dealed with a similar problem? How did you deal with it?
I'm just looking for general tips on how to design this, that's why I'm not giving too much specifics.
I'm asking for help on how to design the general system. The problem is how to define the rules for each notification, given that I must keep an eye on the evolution on the data (the notification isn't defined as "something happened", but as "something happened and kept happening for X time in a row".
The main questions are:
How should I store the rules of the notifications?
How should I check if a new notification must be sent to the user? Querying the database each time I insert data, or is there a more elegant way? Should I keep the state of each possible positive until it becomes an actual notification, or that's an overkill?
Please don't focus on the user settings or if the events should be notified by email or pushed to a mobile app, that's not important :-)