I have to design a system where an API has to be hit to get the meta info about some data. We have multiple workers, that take jobs from multiple queues. Now the API has a rate limit of say 500 per minute.
Since we are using python, so getting the interpreter lock and wait for the time interval won't work as workers here are on different machines.
I have to ensure that workers running on different machines, globally don't exhaust the rate limit of the API.
Few things that affect the design are that this data comes in bursts, like 20,000 30,000 at a time. Multiple workers can pick jobs that process each burst of data and should not exceed the API rate limit.
Research Done :
So, while researching, the most applicable solution I found was to use an implementation of leaky bucket algorithm. Where the system makes requests at constant rate irrespective of the bursts of input.
It is not applicable as it is, because of multiple workers.
Proposed Solution :
Here is a solution that I've come up :
It is a variant on leaky bucket algorithm that also uses a distributed cache lock.
- An input burst comes
- A worker picks the data (say 20,000 items)
- Get a lock by setting a cache key in the cache server.
- Performs leaky bucket algorithm over these 20,000 items by hitting the API at a constant rate within the limit.
When done, releases the cache key lock.
If another bursts comes, while the previous burst was being processed and is picked by a different worker.
- It checks that the distributed cache lock has been acquired, and thus gets added back to the queue with a back-off of 5 minutes.
This way we ensure that only one worker is hitting the API at a time with the allowed rate.
By batching the items, they can be efficiently updated to the data in a single query also.
How to implement the above system in the most robust way possible given the factors? And what can be the improvements in the proposed solution?