I need some advice on an IOT project I am working on. Here are some details about what I am trying to do:
Use Case: Periodically receive data from the 3rd party vendor API in real time and allow users to set alerts when certain conditions are met in their data. Simple example: Send me an alert when temperature goes below 10 degrees. Advanced example: Send me an alert when the temperature stays below 10 degrees for 5 minutes and I am within 200M of location A
Question: What would be some possible architecture solution to my use case? I particularly have no idea how to open the 1k-10k websocket connections on my backend in a scalable way.
- I will be collecting information from 3rd party IOT devices, therefore I will be restricted to the vendors APIs. They currently offer the ability to connect to a websocket to receive the real time data updates. I do not need to share the data with my customers, I just need to alert them.
- I am expecting around 1k - 10k users originally. This could grow but I could not see it ever being more then 50k users.
- Each user will have 1 IOT device.
- The updates will not happen 24/7. I would say about 85% of the day the devices will not be in use and therefore the websocket connections will be idle.
- The data that I am receiving is a flat JSON file with around 200 keys/values. I do not need to store the data historically although I might do so anyways as it could be valuable to my customers in the future
- I already pretty much finished the creation of the frontend/backend which allows users to save the information about their alerts. I used React Native/NodeJS/Express/MongoDB so far. I would like to keep the solution within NodeJS if possible as I am very comfortable using it, but I am open to other stacks if it fits the use case better.
What I have thought I could do so far: My first idea was to simply persist the latest data state in my MongoDB backend and then use MongoDB Change Streams to notify me when the data changes and start the process of checking if an alert condition has been met, however I do not think this would be the most practical way. I have also googled around a bit and heard things like Event Bus, Pub/Sub, Kafka and RabbitMQ as possible ways but really not sure what would be the best fit for my use case.
Thanks for taking the time to read this and help!