0

I have a system which generates pieces of data that needs to be tracked and assigned a specific TTYL value (a value in seconds) for how long they are supposed to live before being re-processed.

I decided to push these messages to mongo db and have a run loop to find the messages that have expired and are ready to be processed.

enter image description here

My concerns: The problem with my approach is that I can see situations where too much load would cause messages to potentially be detected and processed by the run loop potentially several minutes later if things are backed up.

Other considerations:

  • The smallest interval a message can live is 1 second.
  • I don't have to use mongodb but I thought it made sense for this use case.
  • The throughput could be large anywhere from 1 message per second to 10,000 per second

My questions

  • Is this the right approach if not which part of the system should I reconsider?

My attempt

  • I wrote a mongodb query that could potentially find expired messages like so.

enter image description here

enter image description here

1 Answer 1

1

I feel mongodb is not a good fit for the use-case you described, since you have to hand-code the scheduler/run-loop and worry about load and throughput. My understanding is that you'll need a delayed message processor. Perhaps RabbitMQ with the delayed message exchange plugin is a better fit. In short, you can publish the data as a message with ttl (in milliseconds) using the 'x-delay' header. The message will be delivered to your subscribers as soon as the ttl elapses.

1
  • Thanks very much for that input and idea. This does sound like a god idea but any idea if in rabbitmq there is a way to query the queue? Lets say there are times where I want to immediately process all the items even if their ttyl is not over. A piece of information I left out is that each item corresponds to a group_id so there are instances where I would want to immediately process all items for a given group ignoring the ttyl. That's why it became quite complicated. Could rabbitmq somehow be queried like that or maybe combining mongo and rabbit somehow.
    – user391986
    Aug 27, 2020 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.