I need to design an application which makes requests to an external API. External API has a hierarchy of entities: User
which contains ListOfItems
s which contain Items
. I need to get data on all Item
s for all User
s. Within a ListOfItems
requests per Item
can be concurrent/parallel but only one ListOfItems
can be requested at a time, due to external API limitations (ListOfItems
can only be accessed via a token and a token for only one ListOfItems
is issued within a given User
). There will be around 1000 User
s max.
I need to get this data ideally once an hour but in reality this takes longer because the amount of User
s and Item
s is quite large. I already have the code for the method processUser
:
processUser(user) {
List<ListOfItems> lists = getListsPerUser(user);
List<Items> items;
for (list in lists) {
items = getItemsPerListConcurrently(list);
processItems(items);
}
}
Currently, I have a cron job which runs immediately after the previous job finished. The job has a thread executor which creates N
threads and each thread runs processUser
inside a Kubernetes pod. So User
s and Item
s are processed in parallel but ListOfItems
are processed serially. I'd like to scale the solution to multiple pods but this requires to make sure that at any given time same ListOfItems
is not processed by different pods because the token to access user data can only be used for one ListOfItems
at a time (if 2 pods process the same ListOfItems
then we have a problem).
Initially, I thought to go with a message queue as a solution and having each consumer process some User
because there's already a method processUser
and this feature needs to be delivered ASAP. So a message would contain a User
id. The problem is that it can take hours to process a User
. As far as I know many message queues have some interval to acknowledge that a message was processed, if the interval has passed without acknowledgement then the message may be returned to the broker (Kafka has this I think, AWS SQS definitely has this). This means that a message to process User
may be stuck in the queue for hours potentially even a day because all consumers are busy processing other User
s. It feels like processUser
is a long-running task which message queues may not be suited best for.
There're several directions I'm thinking about:
- Use some kind of queue/list of tasks where each task represents a
User
. Spin upN
processUser
instances, then eachprocessUser
instance pops aUser
from the queue. The pop operation should be atomic so that multipleprocessUser
instances don't process sameUser
/ListOfItems
. Maybe I could use RedisBLPOP
which is atomic? This solution would be the quickest to implement in my opinion becauseprocessUser
code is already written. This is good because the feature needs to be delivered ASAP. - Refactor code to use a message queue with ordering guaranteed like Kafka where each message would contain an
Item
id and message key =ListOfItems
id to prevent more than one consumer processing the sameListOfItems
. We already have Kafka infrastructure so just refactorprocessUser
. - Use database synchronization (we use Postgres) so each consumer checks a database table
UsersInProgress
for whichUser
is available and process it. One option is to lock the whole table to make sure that multipleprocessUser
instances don't grab the sameUser
but I think we can just use a transaction where first we check if aUser
is being processed, if not then making a write to the table marking theUser
as being processed. Because writes are atomic we guarantee that only one instance will be able to grab aUser
.
I was wondering if any of the above options sound good taking into consideration that the feature needs to be delivered quickly or maybe there's another option that I didn't think of.
In case option 1 sounds good is Redis/BLPOP really the best solution or there're better solutions?
In case option 3 sounds good will using a transaction to first check if user is processed then mark it as processed work to prevent having multiple instances processing same user?