Design question for handling large volumes of messages in multi-tenant queue

Question

I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario.

Producer produces events based on various inputs (user interactions, api, etc...) and Consumer does down stream processing on these. One of our key constraints is that Consumer can only process events one-at-a-time-per-tenant.

Our current solution (a bit naive) is that multiple worker threads are pulling from the queue and processing events, and if a tenant has another event in progress later worker thread(s) just waits. This has been fine for a couple years given our thread pools and typical event production patterns, but we had a scenario where thousands of events for a single tenant were generated in Producer, and all of Consumer's worker threads except one were stuck waiting. Consumer was therefore processing events from the queue one at a time, and our "eventual consistency" lag time became suboptimal.

We've got some candidate ideas for managing this:

1. Load balancing across queues - new messages go to the most empty queue, but tenants are locked to a single queue (how we achieve this exactly TBD)
2. Create a "slow lane" queue - if during processing of an event, the tenant is already in use, move the events to the "slow lane". This will drain the primary queue quickly but has implications for event processing idempotency I'm not sure will be valid for our scenario.

Before we start digging on these options and looking for others, I'm curious if anyone here has experience with patterns for dealing with this type of situation.

Appreciate any info/advice/guidance. Thanks!!

Answer 1

This sounds like a variation of the 'process events in order' problem.

You can solve the ordering by having a queue per tenant, but with multiple workers you will also need an extra system to 'lock' queues.

Add a "worker.router" program which hands out work to the workers. When a new worker instance starts it contacts the router to say "hi i am ready for work" the router adds it to its list and sends back "please work on queue x". When the worker finishes a message it sends back "finished, please send more work" and the loop continues.

Because the router knows about the list of workers and the list of tenants, it can prevent two workers being assigned to the same queue and detect crashed workers.

You have some hoops to jump through to prevent the router being a single point of failure but its all solvable.

This is a variation of the general pattern where you have the router listen to a single queue and create worker queues for subscribed workers as required. It can hold a buffer of messages to allow reordering and prioritization.

Answer 2

This approach may be viable in scenarios where the message volume is relatively stable for a tenant.

Create N number of queues, start with say 5 queues, and allocate multiple tenants to a queue based on their volume of messages.

For example

500 tenants with a message volume of low       are assigned to queue 1.
200 tenants with a message volume of medium    are assigned to queue 2.
...
  5 tenants with a message volume of very high are assigned to queue 5.

If message distribution is such that each queue averages the same number of messages per tenant, it should even out your eventual consistency time lag.