I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with Faust. This solution would be perfect for my use-case because the work that needs to be done for each message involves a lot of I/O, and therefore the asynchronous nature of the Faust workers would be a good fit.
The problem comes because I also have to be able to schedule messages in the future. I'd like to schedule a new execution as soon as the worker is done with a particular message. The delay depends on the specific message and is not known beforehand.
Furthermore, the queue would contain tens of millions of messages at any given time (messages are small, fortunately, less than 100 bytes usually).
This is what I came up with, but I'm not very satisfied:
- the Kafka queue is initially seeded in some way, and the messages are distributed to the Faust workers;
- when a worker is done with a message, it adds a new version of the message to a Redis sorted set. The score would be the timestamp of the future execution (making it a priority queue);
- a separate worker polls the priority queue at a periodic interval, and takes the messages that need to be processed at that time;
- the messages are sent to the Kafka queue and processed again.
Some more details:
- it's not important that a message scheduled for a certain time is executed exactly at that time. Errors in the range of minutes or even hours sometimes are not problematic;
- messages are not processed in constant time, but could be roughly divided in 2-3 buckets of execution length (and this is known beforehand, from the message);
- if there is nothing to do (which should never happen) then the first messages should be processed anyway, instead of waiting for the scheduled time.
I'm not particularly excited about my solution, as the Redis priority queue could grow too much in size and use up all the memory. It would be best if messages could be rescheduled directly on the Kafka queue, but as far as I know it does not support scheduling.
Final note: I talk about scheduling but the focus is not so much about the exact time, but ensuring that the messages are re-processed (with variants and some messages more frequently than others). If there was a way to reorder messages on a Kafka queue probably it would be a better solution.
I would appreciate any inputs. I hope I made the requirements sufficiently clear.
EDIT: Upon further research, I think I can replace the priority queue with a scheduler, like APScheduler. Such a scheduler supports a variety of backends, including PostgreSQL. Using that I would avoid running out of memory. Each job would simply put the messages on the Kafka queue when it's run. With this architecture, I think I would also solve or greatly attenuate the problems with back/forward pressure on the queue.