I have a system which generates pieces of data that needs to be tracked and assigned a specific TTYL value (a value in seconds) for how long they are supposed to live before being re-processed.
I decided to push these messages to mongo db and have a run loop to find the messages that have expired and are ready to be processed.
My concerns: The problem with my approach is that I can see situations where too much load would cause messages to potentially be detected and processed by the run loop potentially several minutes later if things are backed up.
- The smallest interval a message can live is 1 second.
- I don't have to use mongodb but I thought it made sense for this use case.
- The throughput could be large anywhere from 1 message per second to 10,000 per second
- Is this the right approach if not which part of the system should I reconsider?
- I wrote a mongodb query that could potentially find expired messages like so.