I am working on an application that involves alerts as well as notification policies for multiple users. I am using microservices and am trying to keep them stateless if possible. I have one service that will do the alerting and another that stores and handles policies.
I currently have it partially working by the alerting service gets a message from an SQS queue (that is produced by another service when something fails). My alerting service then asks the policy service what the policy is that is attached to the message coming from the alerts queue.
This policy then contains the number of people attached to the policy as well as the user ID's of the users who are part of the policy and who should be alerted.
Then I loop over the user ID's looking up their specific notification settings. These are then stored into an array.
Part I need to figure out:
I need to figure out how to take the notification setting then alert the correct user via the proper notification method (found in their notification settings I mentioned before). This means that I want to go through each user and notify the correct user using their notification preferences but while not breaking the rules set out in the policy, ie the
escalate_time as well as making sure the order of users alerted is correct. But I also want to escalate to the next user in line once
escalate_time passes for each user.
I was thinking I could add a date/time stamp to the alert but not sure exactly how I would handle a user not 'resolving' an alert.
I have not done the resolving method yet because I am stuck on this alerting part and want to design both parts to work together. I expect it to be a delete request which would delete the alert from the queue(s) it is in.
So in other words, I can figure out how to send the first notification to the first user listed on the policy but I can not figure out how to send the notification a second or third time if they have not resolved the alert.
A bit about my stack, I am using Node.js/Express.js (for HTTP routes) as well as a few AWS services, SQS/SNS/SES. For my database it is MongoDB in a replica set.
Message gotten by alert service from alerts queue
name: 'Polciy name',
escalate_time: 5, // Can be up to 30
Notification part of user model
Note: The values for method can be either email/sms/call and doesn't matter their order or how many are included past 1
Note 2: The contact times can be any value of 10 or under