Architecture for message processing with scheduling, at scale

Question

I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with Faust. This solution would be perfect for my use-case because the work that needs to be done for each message involves a lot of I/O, and therefore the asynchronous nature of the Faust workers would be a good fit.

The problem comes because I also have to be able to schedule messages in the future. I'd like to schedule a new execution as soon as the worker is done with a particular message. The delay depends on the specific message and is not known beforehand.

Furthermore, the queue would contain tens of millions of messages at any given time (messages are small, fortunately, less than 100 bytes usually).

This is what I came up with, but I'm not very satisfied:

1. the Kafka queue is initially seeded in some way, and the messages are distributed to the Faust workers;
2. when a worker is done with a message, it adds a new version of the message to a Redis sorted set. The score would be the timestamp of the future execution (making it a priority queue);
3. a separate worker polls the priority queue at a periodic interval, and takes the messages that need to be processed at that time;
4. the messages are sent to the Kafka queue and processed again.

Some more details:

• it's not important that a message scheduled for a certain time is executed exactly at that time. Errors in the range of minutes or even hours sometimes are not problematic;
• messages are not processed in constant time, but could be roughly divided in 2-3 buckets of execution length (and this is known beforehand, from the message);
• if there is nothing to do (which should never happen) then the first messages should be processed anyway, instead of waiting for the scheduled time.

I'm not particularly excited about my solution, as the Redis priority queue could grow too much in size and use up all the memory. It would be best if messages could be rescheduled directly on the Kafka queue, but as far as I know it does not support scheduling.

Final note: I talk about scheduling but the focus is not so much about the exact time, but ensuring that the messages are re-processed (with variants and some messages more frequently than others). If there was a way to reorder messages on a Kafka queue probably it would be a better solution.

I would appreciate any inputs. I hope I made the requirements sufficiently clear.

EDIT: Upon further research, I think I can replace the priority queue with a scheduler, like APScheduler. Such a scheduler supports a variety of backends, including PostgreSQL. Using that I would avoid running out of memory. Each job would simply put the messages on the Kafka queue when it's run. With this architecture, I think I would also solve or greatly attenuate the problems with back/forward pressure on the queue.

Answer 1

Kafka does not support scheduled delivery, AWS SQS may be an alternative.

Otherwise you have to set up the schedule on consumer side, like you mentioned APScheduler or something similar. Even if you are using Redis or Dynamo db to set up the future timestamp, you can always expire the records after you process them, so storage shouldn't be an issue.

Or maybe you can just use Kafka message scheduler to do this for you with specified timestamp showing when you want the message being delivered.

Answer 2

An alternative to building your own solution would be to use a ready made solution.

There are queuing services that offer built in sheduled delivery. I know of Azure Service Bus. When you send a message or publish an event you can set the ScheduledEnqueueTimeUtc.

Also, a framework like NServiceBus provides scheduled delivery on top of all the transports (queuing systems) that it supports. It will use the native feature of the queueing system if it supports it, otherwise it'll use an implementation based on storage and polling similar to what you mention, but it'll be transparent for you.

Given the amount of messages that you need to handle, I would recomment using some partitioning strategywhatever the solution you choose.