2

We're building a web application to monitor industrial facilities, and we need to add a module to notify the users when certain things happen in the system.

We get data (lots of data) from the facility machines as 10 minutes data series (a value each 10 minutes, like machine's components temperatures, output performance, power consumption, etc), and we store the data in a cassandra DB. The framework used is Django.

These are the kind of things that we need to notify (and we need to emit the notification when the event happens, so no batch processing):

  • When a machine has been under 90% of its theoretical performance for 60 minutes in a row (i.e. 6 rows of 10-min data), but not if the machine has been manually stopped.

  • When the whole facility has been under 95% of it theoretical performance for two hours in a row.

  • When a machine is manually stopped.

  • When the monthly production reaches 50% of the expected.

  • Etc, etc.

The details of the notification (percentages, time spans, etc) should be configurable per user.

We are looking for a design as generic as posible, because the list of available notifications will grow in the future. Has somebody dealed with a similar problem? How did you deal with it?

I'm just looking for general tips on how to design this, that's why I'm not giving too much specifics.

UPDATE

I'm asking for help on how to design the general system. The problem is how to define the rules for each notification, given that I must keep an eye on the evolution on the data (the notification isn't defined as "something happened", but as "something happened and kept happening for X time in a row".

The main questions are:

  • How should I store the rules of the notifications?

  • How should I check if a new notification must be sent to the user? Querying the database each time I insert data, or is there a more elegant way? Should I keep the state of each possible positive until it becomes an actual notification, or that's an overkill?

Please don't focus on the user settings or if the events should be notified by email or pushed to a mobile app, that's not important :-)

  • 1
    I suggest you have a look at how a tool like Nagios works. Not that it's exactly what you're looking for, but you may find very interesting concepts here. I'm thinking of e.g. mapping performance counters to a common scale (ok, warning, critical) across all components. Also, per-user settings seems a complicated task. Are you sure your users (who are they btw?) are actually going to configure anything? – Tibo Sep 29 '17 at 18:40
  • I agree, per-user event notifications does not seem a feasible approach for this, since you have lots of data. Can you provide more information about how the user will receive the notification? It must log into the web app and check for them? – Emerson Cardoso Sep 29 '17 at 19:14
  • 1
    You do not want to start with generic design from get go. As you don't properly understand the problem domain. You should start with concrete solution to concrete problem for the first version. If it succeeds, you can implement second, more generic, version. This time, with better understanding of what problems you have. Especially we won't be able to help you, as we have no understanding of data, processes or scale of your problem. – Euphoric Sep 29 '17 at 20:39
  • 1
    The general field for this sort of thing is called complex event processing. – Derek Elkins Sep 29 '17 at 21:35
  • This type of monitoring is often built into operating systems. You may wish to see how they are designed. – Frank Hileman Sep 29 '17 at 21:42
1

So, I'll say up front it seems like you may have gone down the "Let's build it from scratch" path without doing sufficient research. The kind of system you want exists in a lot of places, many of which aren't even requiring you to pay.

My favorite tool for this kind of problem is the Elastic Stack. It's highly flexible and highly scalable for taking in a ton of events with low latency, parsing them and indexing them. There are all kinds of outputs that you can configure to include notifications.

There's even a specific article they publish on doing alerts and notifications based on complex events..

  • I've dome some research, but, as far as I didn't know exactly what I'm looking for, I didn't find anything relevant. But that article seems very promising! We will study it and come back to accept your answer if it's useful to us. Thank you very much! – César García Tapia Sep 29 '17 at 22:00
0

I would do the following:

  • Create a language for the rules. This does not mean you must create a text based language, with a parser. It means you think about the operators needed to compose rules (they are composite objects), as well as the terminals (leaf nodes of expressions), and create a library that contains an object for each component of the language. Now you can compose rules.
  • Rules need an evaluate or execute method that traverses the object tree and determines if a notification is in order. The simplest is a method that returns a boolean.
  • Rules need to be persisted using your favorite technique. I would use binary persistence, but you can use whatever you fancy.
  • I would keep the rules in memory in a server process. I would keep the smaller bits of input data (data supplied to rule evaluator via a context object) in memory as possible as well.
  • If you have much real-time data, I assume you have thought about performance already. If not, consider simple binary streams in a file system; whatever you do, don't do the equivalent of a row in a database per incremental bit of data coming in.
  • The incoming data is probably read-only so you can get away with creating new files per chunk of data, and cleaning them periodically with a garbage collector.
  • If you are concerned about the overhead of evaluating all rules each time you get new data, you could keep a list of input data used by each rule at the root node of the rule, computed once after rule creation. Use that to determine which rules are affected.

Of course there are lots of optimization strategies; measure before optimizing anything.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.