We are trying to redesign our microservice architecture-based application to support multiple tenants. We have a simple queue service that will be utilized by other services to queue any asynchronous jobs. On the other side individual microservices will poll the queue service to fetch/pick any asynchronous jobs that they can process at frequent intervals (dequeue).
To ensure fairness during dequeuing we are using a simple round-robin algorithm implemented using a circular list in Redis (https://redis.io/commands/lmove#pattern-circular-list). We maintain the service name as the key/list name and all the tenant identifiers as value. This allows us to do a round-robin on a per-service basis whenever a fetch request lands from a particular service. We have used Redisson's Deque data structure (which stores all the Tenants) to iterate over all the Tenants in a round-robin manner.
This approach has a disadvantage. Even the tenants with no jobs (which in our case is higher) are being considered when a fetch request lands and this results in some delay for tenants with jobs.
Instead of adding all the tenant identifiers to the list, we can add only the tenants with jobs as we can identify the tenants when the queueing operation happens. We can remove the tenants from the list during dequeuing. But since our application is multi-threaded we are not so sure on how to handle the concurrency part in Redis i.e there is a possibility that we could remove a tenant from the list just as he gets a new job.
For example, tenant1 was added to the list when a job was queued and during dequeuing just as we issued the command to Redis to remove tenant1 from the list another new job for the tenant1 landed. Now during this new queueing operation for the new job, we will try to add tenant1 back to the list. Due to network latency or the multithreaded nature of the application if the add tenant operation got executed before the remove command and then the remove command would delete the tenant identifier from the list. This would result in tenant1 not being available on the list even though he has a job in the queue.
How can we solve this issue or are there any other approaches that we can consider?