2

I am building a solution where IoT-devices send in measurement-data to an API that's hosted on AWS. For each measurement-type the user can set a threshold value; that when reached will trigger an alert to be sent out.

My design is based on events, so each new measurement that's received in the API generates a work item on an AWS SQS-queue. A lambda-function then processes the work-item and reads all the threshold-alerts for that particular device from a database, and checks if any of the received data has passed the threshold. If the threshold is passed, it sends out an alert email.

The API processes about 1000 requests each hour and the reading of all threshold-alerts from the database is getting expensive/time consuming.

So my question is if there is a better way to design this alert system? I was thinking of adding a cache-layer that will cache all threshold-limits since they don't change that often, but this means I have to use a distributed cache and still make a roundtrip via HTTP.

All suggestions welcome!

System workflow

|improve this question
  • If you use e-mail to signal alerts, your timing requirements are probably quite relaxed. This means you can buffer events for say a minute and query the database for alerts concerning just the 15-20 events that were received in the last minute. – Hans-Martin Mosner Apr 10 at 12:01
  • 3
    Do you know where your bottleneck is? 1000 events/hr isn’t a lot. – RubberDuck Apr 10 at 21:17
  • Leave each node to load the thresholds into memory (local cache) at least once during the bootstraping. Evict the local cache frequently (every X hours, mins, whatever). I don't see a need for a distributed cache. If you do, use systems like REDIS. A single instance is going to be more than enough. – Laiv Apr 11 at 7:02
  • Thanks for all your replies. The reason I want to redesign is because the volume is increasing each month. For each event that is recieved I hit the database to get alerts for that particular device, which is going to be wasteful if the volume keeps increasing. Buffering events and batching might be a good idea. – klas mack Apr 11 at 8:00
  • Do you have control over the devices making the measurements? Can they be programmed to do the threshold computation themselves? – Blrfl Apr 11 at 12:47
1

The best way to do this is not to query the data, but assess it as it comes in.

So have multiple worker process applications, each with an alert rule all listening to events as they come in from the IoT devices in a fan out queue.

Each event is processed to see if it hits the criteria for an alert and if it does the worker process raises an alert with some alerting api.

IoT Device -> messages
messages -> incoming queue
incoming queue -> fan out to RuleAQueue, RuleBQueue etc
RuleXQueue -> RuleXWorker
RuleXWorker -> check rule and raise alert if required.

In this way we ensure that we can horizontally scale the rule checking if required. Either running all the rules on the same box, or having a million boxes with one rule on each.

We remove the bottleneck of having to select the data from a database.

In your case with only 1000 actual alerts per hour, it sounds to me like it's querying the db for the data, presumably every few seconds * 1000, rather than evaluating each test which is the performance bottleneck. So you can probably run all the workers on the same box, or even combine them into one app.

|improve this answer
  • Thanks for your reply! Do you mean creating 1 lambda-function for each alert set up? In this case, a user might have 4~ alerts set up, and if I have a 1000 users that means I need to keep check and update 4000 lambdas. A alert-rule might be something like: 'If measurement X is > 70" and is unique per user since they might be more customizable in the future. – klas mack Apr 10 at 10:20
  • no. A. dont use lambas, but thats a different conv. B. one per alert type or rule. I need more context for an example – Ewan Apr 10 at 10:38
  • Context for alerts: User logs in User registers IoT-device Sets up a measurement-capability of type Temperature for the device Sets up a alert that "if Temperature > 25 celsius for this device send me an email" Starts sending measurements to API and expects alerts to work. – klas mack Apr 10 at 10:41
  • OK so either : A add the alert criteria to the message and have 1 worker for all temp messages. (alert if currenttemp > alertTemp) B. shard devices by temp and route accordingly (i imagine there is a small range of temps) worker for 25, worker for 26 etc. C: shard by user or user group so you can hold their target temps in memory. worker for Users A-G, worker for users H-K etc – Ewan Apr 10 at 10:46
  • maybe you can hold all the deviceIds and target temps in memory and just have the one sevice? – Ewan Apr 10 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.