4

I have a system running in Azure that accepts messages via a REST endpoint. The customer has a certain number of devices that send some data for example temperature and we present this data in a web interface. The part of the system that accepts messages is separate from the website (microservices!) but they access the same database (not so microservices?). I am building a new functionality that allows the customer to setup notifications. For example the customer defines that if any of his devices sends a temperature above 50°C then an e-mail should be sent. The notifications can be instant (or of relatively short period) or aggregated (daily, monthly, etc.).

The way I had built these things in the past on premise was to setup a Windows Service that polls the database periodically gets the notification definitions, gets the corresponding messages and sends the proper e-mails be it instant or aggregated. However something tells me that this way is not cloudy at all. Am I supposed to use a message queue, a service bus or something else? If yes which one? What about aggregated notifications? If not what is the best equivalent of a Windows Service in Azure today for my use case? Am I still supposed to use a Worker Role?

Also note that in the future the messages may be taken out of the relational database and put in a NoSQL solution and in practice they are much more complex than one number (the REST endpoint does some processing before storing them in the DB). The REST endpoint does have access to the notification definitions and could potentially process notifications itself but I'd like to avoid that because I like the clean separation and simplicity of the separate services I currently have. Also processing the message when received does not solve the aggregation problem. In case it matters we expect relatively small number of users but great number of messages and hopefully great number of devices sending messages. In case it is not clear there are multiple users each of them getting notification for his own data.

  • Not answering the question directly, this sounds like you are reinventing nagios, naemon, op5, any-other-of-the-myriads-of-monitoring-systems? – Mattias Åslund Apr 22 '16 at 5:07
  • Honestly I never heard of any of these. Will check. We actually implemented it the stupid way and decided that we will do something to scale it when we need to (we have a plan). I will write a detailed answer one of these days. – Stilgar Apr 22 '16 at 5:55
  • @MattiasÅslund I checked these tools. They seem to be designed to monitor the health of the system itself rather than as a mechanism for alerts. – Stilgar Apr 22 '16 at 7:50
  • Monitoring systems support passive and/or active checks, where almost any resource (db, filesystem, website, switch, your-proprietory-machine) can be checked. Measured values are compared to min/max ranges and when a resource has been sufficiently out of control some action is taken. Usually that means the system sends an sms/email to a group of operators, based on a time schedule. If the problem isnt acknowledged/solved in an appropriate time the alert can be further raised to another set of admins. – Mattias Åslund Apr 22 '16 at 14:09
  • Yeah but in this case the e-mail, phone number, etc. must come from the system itself. The user defines his own notifications. – Stilgar Apr 22 '16 at 14:24
0

We ended up implementing the "Windows Service" scenario knowing that it would not scale but intending to build a scalable solution when we have to. With the current implementation a Web Job runs a console application that queries the DB for all notification definitions, runs appropriate queries and sends e-mails. It is easy to test (just run the console app) and easy to deploy (web jobs deploy with the web site). The best part is that if we go for the scalable solution we can just remove this part without affecting any other code because notification definition UI and database schema do not depend on the actual sending method.

We ended up deciding against Stream Analytics because Stream Analytics are not very useful for ever changing queries. Changing the query requires a restarting the Stream Analytics job. Putting the notification definitions in the job is hard. There is something called training data that can be used but it is not very suitable for this case.

After we implemented the solution Microsoft launched Azure Functions which at first sight seem like a good fit. I did not do a detailed research because we already had a working solution at that point. I tend to dislike how separate Functions tend to be from the rest of the codebase and I worry about deployment. I guess there are solutions to these problems but they were not in the demos.

The scalable solution that we had in mind and will evaluate again if we need to scale would work as follows - The Message processing part sends each and every message into azure queue. A consumer gets the messages one by one and checks if the message is eligible for any notification. If it is it is put in DocumentDB or Azure Tables bucket list which is accessible based on the notification ID(s). Another consumer periodically checks the notification buckets and sends all the messages and deletes the bucket list. The solution seems scalable because we can add whatever number of consumers we need and azure queues are built to be scalable. There seems to be no bottleneck part. Now of course with Azure Functions we can replace parts of this systems with if they turn out to be better suited for the task.

Of course I am still open to criticism and other ideas.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.